Then I had an evening dinner with Jen, and a weekend. I'll write more on that later, I hope.
Monday was mildly chaotic, I came in early and worked on digging myself out of the massive hole of workload I'm in because of my being out of the office so much (*gripe about stupid non-trivial travel for no good reason here*). I actually managed to get mostly caught up, and even had some (gasp!) spare time in the afternoon, which gave me time to noodle around with some SNMP stuff and discover a major misconfiguration on our WAN's edge routers at all the sites. Keen! So I spend an hour and a half touching each router remotely and repairing the misconfiguration. During that time I hit upon a stroke of genius, and in my head architected a system to automate a disaster recovery process as well as create a methodology to back up the software and configurations of all our WAN routers. Now I just have to find my Perl book and take the time to write it up. Let's see if I can manage to keep things rolling enough that I still have "open time" to work on this extra-credit project.
Monday night was not good, as I received a call at 10:15 in the evening from our Helpless Desk manager asking what was wrong with the firewall. Two hours (and only one answer to my multitude of questions) later, I managed to get to bed... to come in to total and complete chaos at the office this morning, because our firewalls were both hosed. I'll save the detailed description for later. Right now I'm still trying to catch my breath, get a bite to eat, and then I go back down to the NOC to plunge headfirst back into this problem along with our contracted support team. I'm skipping my scheduled meeting to ensure that I get this thing fixed, because I do not want this problem to persist any longer than it already has. It's reached epic-stupid proportions already.
Wish me luck.