I've been tagged at least twice now (Denis Gobo and K. Brian Kelley) for the Toughest Career Challenge.
My story: I took a job as a contractor, temp-to-perm. It was a relatively small shop and I was the only person working with databases. Since the shop was small I wore other hats. One of them was maintaining front-ends (forms and web clients) for financial applications.
The "to-perm" part never materialized. The first clue was about six weeks into the gig. I was responsible for implementing an upgrade. The schedule was set before I was hired and could not be altered. This had all the markings of a Crappy Job Test.
Crappy Job Testing is important. You can use CJT to detect whether you have a Crappy Job. A good Crappy Job Test goes something like this: Early on you're given or decide to tackle something big. You dive in and succeed - or fail (you can learn from both). If you fail and are fired, that was a Crappy Job. Getting fired from a Crappy Job is a lot like getting thrown off a sinking ship: you're going in the drink eventually, might as well get it over with. If you succeed and nothing changes - there's no trust or respect (or no more) demonstrated by your boss(es) or peers - you have a Crappy Job.
I succeeded in the upgrade. The software was troublesome, buggy, hard to install, and even required hacking to set one of the passwords (the web interface was broken). But I persevered, overcame, and adapted.
If anything, I was treated with less respect after the success. I began a quiet job search the very next week.
I still did a good job. I came in early and worked on call late nights. I introduced virtualization and the Scrum methodology. Projects came and went, and life went on as normal for a few months. And then one Monday morning something happened.
We got a call that someone couldn't access a project in the project management application. I looked and, sure enough, that project showed deactivated. Then another call came in. Same problem - couldn't access the project. I checked and - you won't believe this - that project was also deactivated. I decided to execute a query on the database to see where we stood on active vs. inactive projects. There was some history in there, but about 10% of the projects should show in an active status. There were 140,000 projects in the database. 75 were active. And all of them were active since late Friday afternoon.
My boss asked me how this could have happened. I told her I didn't know, but all the inactive records in the table showed the same last-updated date and time: late Friday afternoon. She asked for theories, I had none. She asked if I could be responsible and I said "Yes". She asked about details, I told her this could happen if I or anyone with access to the database (there were a few of us) executed an update on the table and forgot the Where clause.
She became physically upset. I found the last backup and began restoring to a new database on the test server. I'm one of those people who doesn't do reactions. Unfortunately, I was working for someone who didn't think they were getting through to you until you reacted. It was a classic lose/lose scenario.
After several hours of troubleshooting, I discovered a massive update had in fact occurred late Friday afternoon - at a time when I was about an hour into a three-hour road trip. I had identified the offending T-SQL embedded in a large stored procedure.
The oddest thing happened when I reported it wasn't my fault and I could prove it: the tension melted. It turned out, as of that moment, that it wasn't so big a deal after all.
Now if I was paranoid, I'd say my boss was gunning for me. It could've heightened my paranoia when she introduced me to people over the next few days that were interviewing for the "database position". Or when my job description showed up on a large jobs website.
In the end everything worked out for the best. I was in the late stages of interviewing with three companies by that time, having started the job search when I realized I had a Crappy Job. I accepted an offer and submitted a notice a couple days before I was to be "let go".
It wasn't so much a technical challenge as a business one - a personality clash is more accurate. It was a disappointing experience, but also educational. There's little you can do if you irritate someone by merely being who you are. From this experience, I learned the most important words in the phrase "I have a problem with you(r)..." are the first four.