Wednesday, September 12, 2007

Another Day in Paradise

This past week or so has been basic hell at my work. And I don't mean due to unreasonable work loads, unreasonable management, or unreasonable expectations. No, this past week was unlucky engineering hell. So many things went wrong that you'd think we were cursed or something. Maybe we collectively have enough disbelief in god that we were put through the wrath of hell as punishment. I'm not really sure. But as my boss said "things are really falling apart." That might be an understatement.

Collectively, in the past week the following things occurred.

We lost 2 development servers, including our source control, bug management software, wiki and other resources running from these machines. Of course we had backups and we were able to reassembly our development lives on another machine. But that took many, many hours and we weren't complete recovered several days later when, BAM!, it happened again. The second time it happened was the last day we had to work before a very large deadline.

To those of you less technical, this situation is the equivalent of being a good driver and blowing out three tires on the same wheel, each occurring about 20 minutes after the previous one had been replaced. It's bad and unlucky. It's probably statistically worst than unlucky. It's down right weird.

We had another situation that I won't get into here, that essentially caused all of our account permissions to be changed in such a drastic way that I couldn't get on the servers I needed to be on to do my work. This caused a large waste of time for all of our developers as we waited and worked towards resolution.

We had a huge deadline late last night and I got up especially early (before 7am) to get to work early so I could possibly not work all night long. I was there for 12 hours before I could do any work because of the second development server that was lost, which I already mentioned above. 12 freakin hours!! Then finally, about dinner time, we were back in business for work. It sucked. The worst thing about it was knowing this huge deadline was so near, and not being able to work towards it. I worked until about 2AM until I was so tired that I wasn't programming well anymore.

Then, lucky me (but not my other co-workers) I was so tired that I slept in this morning and showed up to work about noon. This being just mere minutes before the internet at our office was repaired, because it had been down all morning. So, most people in our office couldn't work. Did I mention that our phone system is also on the internet? There were no phones either. It was a rough day for the whole office.

And! The cream on the cake was that while we had no internet in our office, Google maps changed their API which resulted in the maps on our live website to break. But, we didn't have any internet so it was near impossible to fix. It was fixed by breaking every programming rule in the book. We edited live code on live web servers. Scary. Did I mention how scary that is yet? Well, sometimes you got to do what you got to do.

There are a couple more happenings that belong in this pile of unbelievable bad luck, but the mundane details aren't really worth your time. All you need to know is that I might be cursed so pary to god before you see me next.

In other news, the yard that we're working on clearing out the vines from is doing well. In no time at all we'll be able to build some food gardening beds and start growing our own food. I'm looking forward to that. Also, this past weekend punkinpie and I got together and prototyped custom homemade blinds for our living room windows. I did the sewing and he did the wood working and it came out really cool. Did I mention that their top down bottom up ones?

No comments: