I should have known that a three day weekend meant there was an extra day for things to break at work, but I was completely taken by surprise at just how broken things were when I actually got into the office on Tuesday morning. The first sign of trouble came to me as I walked in the door... RDY, one of the leads on a major software project for the company, was coming out to take a smoke break. As he passed me on the steps he told me in a casual voice, "Oh, by the way... OSS-2 is broken." Normally I don't deal with the servers, the Oracle software or the applications -- I'm just supposed to be a network geek, so I sort of ignored it and filed it away under "alright, that's good to know." When I sat down at my desk I had two urgent tickets, one telling me that OSS-2 was down (that wasn't entirely unexpected, but I still didn't understand how that was any of my problem) and one telling me that the Miramar, Florida campus had taken a lightning strike and fried a bunch of their network gear. Not much I could do about that, so why did I have this ticket? I scrolled down a little further... aaaah, there's the answer: They'd gotten the ILEC out to repair the damage (that must have been some support call to garner such a swift response.... aaah, they'd burned out two smart jacks, fried over 18 pairs from the CEV and lost a distribution switch in the deal. Must have been some lightning storm!) and were having problems with their Voice-over-IP gateway... the primary had failed during the surge and even after being rebooted it refused to synchronize with the PRI again. They'd "failed over" to a secondary VoIP gateway that had been set up just for this very possibility and left unplugged. The good news is that the campus was up and running... but they needed somebody to check the failed unit to see if the CSU was indeed dead.
I made some calls, starting checking into things.... and then all hell broke loose. Not less than twenty minutes after I got logged into the help desk I was barraged with tickets. Classroom Six couldn't reach any of the resources that they could reach last week, but the servers were all reachable via ICMP (this made issue number three). Another campus (Galleria, GA) was down for reasons unknown, marking issue number four. Another Oracle application was reachable... but whenever somebody logged in it would present them with a blank page -- issue number five. Of course every single one of these tickets was technically "urgent" since it was impacting business-critical work for a number of users. I spent the rest of my day juggling one problem after the next, constantly on the move, never getting a chance to really sit down and take a breather. The most frustrating thing for me was that at the end of the day only one problem was something I got resolved to my satisfaction through my own direct efforts: I was able to troubleshoot the Classroom Six issue and, with a simple NAT rule on the firewall, achieved a resolution. Galleria was down because the ILEC had come in to do some work for another building tenant and had apparently stolen the bridge clip for our T-1 span off the 66-block. Thanks a lot, guys! OSS-2 wasn't reachable if you were on the 10.2.2.0/24 network because of an architectural oversight on the half of my supervisor, an oversight that only caused problems once he instituted a cutover.... which he'd done this Sunday, which was partly why we had so much chaos on our hands. Miramar eventually got some of my time (I put them low in my queue becase although they were running in a "degraded" status on their backup VG they at least had a functioning campus) and I sat down with the guys on the VoIP team to learn a bit more about the device in question. This was a good choice on my part: I walked away with about a page worth of notes, and handed the work off to BB, the same coworker who'd accompanied me to Southern California. I told BB that at the very least the CSU was toast, the module was more than likely fried and if we were very unlucky the entire backplane was crisped.... so Cisco better get moving to the site with a replacement. When he asked how I'd reached that conclusion I just smiled... the VoIP team showed me how to read the logs, and I saw that the DSP was reporting continuous errors, a sure sign it had been melted by the voltage spike on the line.
Sometime around 4 PM Tuesday afternoon I was able to escape the madhouse that was my office. I got in my truck, started it and grimaced at the horrible noises that erupted from the unmuffled exhaust system. I drove home and stopped by AutoZone on the way to pick up some parts. I knew I needed a new air filter, and figured I might as well replace my PCV valve and the crankcase breather while I was at it. Another item I purchased was a new pressure cap for my radiator -- I thought that with luck it would be the key to my engine temperature problem. I also picked up three exhaust patch kits (two of one type that looked promising and were dirt-cheap, and a third of a different more expensive type in case the first two failed). I got home, left the truck to sit and watched some TV for a while. Dinner for myself consisted of cheese and beef sticks, that old Atkins standby. Around 7 PM I decided the engine had cooled down enough for me to commence my repair work on the exhaust pipes. I took some of the patch material from the first cheap kit, cut it in half and applied it to my exhaust system... then I did the other half of the y-pipe. I should have used both kits at once but I thought that I'd be able to get the work done with one kit. I actually did, but then discovered that I was short of the wire provided by the kit to keep the new muffler patch tightly wrapped about the pipe. I didn't think it was a problem though, and went on a fifteen minute drive as specified in the directions. Just as the product packaging promised the tape heated up, the material changed to liquid and flowed together, then solidified after I let the engine cool. The downside is that only half of my work took.... and guess which half it was? That's right, the half with the wire... because it was held tightly to the exhaust pipe. The other half melted just as the first half had, but without the wire to keep it in place it had formed not so much a patch as it had a loose baffle. I cut the malformed patch free and left the other one to cool, planning to complete the work with the remaining cheap kit on Wednesday afternoon since it looked like the product did actually deliver on its promise to fix the problem.
When I started the truck this morning I was pleasantly surprised to hear how much more quiet it had gotten, even with only half the work completed. Naturally there was still some noise and rattling, but it wasn't nearly as bad as it had been earlier in the week (Roho can well attest to this). I drove down to work and, once again, had a busy day. Fortunately for me it wasn't as hectic as Tuesday was, so I had a little more time to breathe and plan out my course of action for the varous problems that I was faced with. Most of my time was spent troubleshooting the issue about web pages being displayed as blanks, and just as I was getting set to leave the office this afternoon it looked like we might be approaching a fix for the problem. Unsurprisingly the "problem," and the "fix" for it didn't seem to be network related... it looked like it was entirely an application-side issue. I am Jack's utter lack of surprise at this... but I won't know for absolute certain that it's a problem with the DBA's software until I check in tomorrow for a status update. That reminds me, I won't have any sort of status waiting for me when I get in, I'll have to brave one of the DBA's cubes and ask them directly to find out what's going on. I really need to hammer home the importance of communication to my supervisor... at least with regards to high-profile problems like this....
Leaving work was quite an adventure today, too. At 3:54, just as I was getting ready to call it a day, I stood up and looked out the window. Call it a day? I should have called it a night! It was pitch-black outside, and even from where my cube was I could hear rain pelting against the glass wall that comprises the exterior of our office building. If it was that dark out, and I could hear the rain I knew it was one heck of a storm. I decided against trying to bring my laptop home with me and instead wrapped myself up in my coat, grinning from ear to ear. Each summer I catch all sorts of flack from my coworkers for wearing my black trench coat in to the office. Everyone asks me how it is I don't get overheated and pass out, or they want to know if I have weapons under there like that scene in "The Matrix," or if I have my government ID. They can laugh all they like, though, because I am smug and confident for this reason: while people were desperately searching for an umbrella they didn't bring with I had on a nice waterproof microfiber coat that I knew would keep me dry no matter how bad the storm I was about to step into.
Walking outside was interesting to say the least. The parking lot was flooded from the sudden downpour we'd had minutes before, and the office management never cleans the drains that serve the 'lot so they were of course completely backed up. Above me the clouds were an ominous shade of greenish-black, thunder rolled in the distance and lightning flashed directly overhead. What worried me the most, though, was something else entirely: as soon as I stepped out of the lobby's revolving door I heard the tornado sirens wailing all around me, indicating that the severe weather was by no means over despite the fact that the driving rain had lessened somewhat.
I made my way to my truck, gave my coworker a curt nod and dove into the cab. I wasn't at all happy about being outside until I was six miles away from the office and making my way onto the freeway... which had turned into (surprise!) a complete and utter parking lot. This state amazes me... just add water from the sky and you get six million idiots, all of whom feel strangely compelled to cram themselves onto the major thoroughfairs so they can drive along at a snail's pace. I drove over 18 miles at 30 miles an hour... just because the sky had gotten cloudy and water had fallen from it! No other reason! WAUGH!
I guess maybe some of the drivers were perplexed by something other than the water-from-the-sky, since the roads had changed a bit today. The construction on I-290 has gotten about one-quarter completed, and with this milestone they shifted the westbound traffic onto the new concrete paving. Truly it is amazing that we can remove the major obstacles everyone had to face on the detour (the twists, the shuffling, the narrow lanes, the distraction of having oncoming traffic so close you could reach over the concrete barrier and touch it if you didn't mind losing a hand for your troubles) but they still went slowly. Very. Slowly. Maybe it's because they wanted to stare at the bright new concrete they were driving on, or maybe after all this time they were used to the detour and were now intimidated by the concept of going "straight forward." Whatever their reasons.... 30 miles per hour the entire way, with lots of stop-and-go for good measure. It didn't stop even when we got past the construction zone! The end result? What should have taken me 38 minutes at most was a 54-minute ordeal... but I got lucky, I think. I can't imagine what the drive would have been like had I left at 5'oclock to catch the "normal" rush-hour.
When I got home most of the rain had stopped and the air was moist, but not oppressively muggy because of the low temperature. Since the rain had made the lawn muddy I walked on the sidewalk the whole way to the front door, and it's a good thing I did too... I got to see the damage Ra had done to the screen that guards one of my windows. He's worn a hole in it of just the right size so that if he were so inclined he could slip through and thus escape the confines of the apartment. This is of course a completely unacceptable situation, one I remedied by I closing the windows as soon as I got into the bedroom. I've decided that Ra can't be left unsupervised around ANYTHING with a screen since he keeps trying to go through it. To make matters worse this isn't the first time he's worn a hole into a screen -- and the last time he did it the little punk took a three-story fall so that he could disappear for the better part of three months, leaving me to tear my hair out.
No... I don't think I'll be letting that happen again.
So having seen what damage he's done I've got another item to add to my weekend list of chores. Now, in addition to everything else I'll have to take the screen out of the window, carry it over to True Value and let them patch it so that we have no further escape attempts. This will also have the added benefit of preventing us from losing any money on the damage deposit or the apartment.
With the rain outside and the soggy ground I didn't feel very inclined to climb under the cab of my pickup again, so I opted instead to stay inside and shoot the breeze online. Around 7 PM I shrugged out of my work clothes and into my "exercise" clothes, namely a pair of shorts and an old and faded t-shirt from my days at Holiday Sports. I dropped "Demolition Man" into the DVD player, put on my sneakers and climbed onto the treadmill for some much-needed exercise. I figured that since it's been over 3 weeks since I put any time in on the belt I was due for a good workout, and I fully expected to be in a great deal of pain for it. Pleasant surprises were awaiting me, it seems. I discovered that with the new configuration of the apartment I had a perfectly unobstructed view of the TV and was able to watch the movie for the entire duration of my walk. I put in over 30 minutes on the treadmill, burning 229 calories and walking more than 1.75 miles... and it didn't feel like it had taken me more than ten minutes to do it! This is a far cry from the old days of staring at the wall and begging for the torture to be over. It seems that so long as I have a distraction I hardly even notice the change in the treadmill's incline nor the passage of time. Another thing I didn't notice was just hot and sweaty I was towards the end of the routine. I discovered for myself quickly enough once I turned off the machine and stepped down! I also realized within moments just how offensive I was and changed into my robe, then made a quick trip to the bathroom for a cool shower and a good scrubbing.
I've spent the rest of the evening with a pleasant feeling of being clean, exercised and well-fed. I hope that I'll be able to add "well-rested" to the list of accomplishments when I get up tomorrow and reflect on today. With luck on Thursday the weather will be clement enough for me to finish wrapping the other half of the y-pipe so that I can breathe a little more easy when it comes to the exhaust issue (and I mean "breath easier" quite literally... having exhaust gasses venting directly under the cab is a nasty, dangerous thing). Once I know that problem is resolved I'll go back to playing the guessing game of "Why is the engine coolant running so hot?" I think I may have the winning answer at this point and I don't like it, not one bit. The cost is not acceptable, the "downtime" I'll be faced with is definately not acceptable from my standpoint (It would infringe on my paid vacation) and it's silly especially since the new truck will be ready in just over a month if all goes well.
I forgot to mention above: captain18 left Tuesday with his fiancee spoothbrush for their new home in Binghampton, New York. I fondly wish them both a safe drive, and look forward to attending the wedding in August. And Cap? In case you didn't know... you're going to be missed around here, my man. You've been a great friend, and I'm deeply sorry that I didn't get to attend your dinner on Sunday to see you off and give you the official well-wishes. I know you're going to make a great husband and you'll do a kickass job in the broadcast industry out East.
My wrists are starting to ache again, so I think I'll let this entry draw to a close.
Would I still see suspicion in your eyes?