April 9th, 2002


Short update...

Worked late Friday, as our Fremont campus decided on the spur of the moment they needed to get off their (malfunctioning) firewall and onto ours. Funny, I'd been pushing that for six months -- but they never seemed to care until their box went tits-up. Then it was a national emergency and I had to get it done right then, despite the lack of planning, diagramming or documentation. I am not the kind of guy who needs a project plan with accompanying Gaant chart to brush my teeth, but I don't like doing major network alterations without some sort of idea what we're doing, to what, why and when. So I spent four production hours doing the cut over, and an additional three and a half trying to clean up the mess that I was faced with when the inevitable implosion occured.

Then I had an evening dinner with Jen, and a weekend. I'll write more on that later, I hope.

Monday was mildly chaotic, I came in early and worked on digging myself out of the massive hole of workload I'm in because of my being out of the office so much (*gripe about stupid non-trivial travel for no good reason here*). I actually managed to get mostly caught up, and even had some (gasp!) spare time in the afternoon, which gave me time to noodle around with some SNMP stuff and discover a major misconfiguration on our WAN's edge routers at all the sites. Keen! So I spend an hour and a half touching each router remotely and repairing the misconfiguration. During that time I hit upon a stroke of genius, and in my head architected a system to automate a disaster recovery process as well as create a methodology to back up the software and configurations of all our WAN routers. Now I just have to find my Perl book and take the time to write it up. Let's see if I can manage to keep things rolling enough that I still have "open time" to work on this extra-credit project.

Monday night was not good, as I received a call at 10:15 in the evening from our Helpless Desk manager asking what was wrong with the firewall. Two hours (and only one answer to my multitude of questions) later, I managed to get to bed... to come in to total and complete chaos at the office this morning, because our firewalls were both hosed. I'll save the detailed description for later. Right now I'm still trying to catch my breath, get a bite to eat, and then I go back down to the NOC to plunge headfirst back into this problem along with our contracted support team. I'm skipping my scheduled meeting to ensure that I get this thing fixed, because I do not want this problem to persist any longer than it already has. It's reached epic-stupid proportions already.

Wish me luck.
  • Current Music
    Keyboards clicking