It's 1:15 in the morning on Wednesday and I just got home from work. Due to a network change on the 27th floor (migrating users from a cluster of 3524-IP switches to a 4006 switch) I had to work the 5pm to 1AM shift. The last time this change was attempted I wasn't involved, but apparently the 4006 went unresponsive and essentially crashed for reasons unknown. As if that wasn't bad enough when the switch did come back up none of the IP phones on the floor would register with the Call Manager. Thankfully tonight went a little bit better. We got the work done earlier than expected (We had planned on running to 1 AM, but we got done about 12:30) so I excused myself for the night and left "early" as a reward..

Like most not-quite-major-but-certainly-not-minor changes we would have gotten done about two hours earlier if there hadn't been a sneaky problem that required some time to troubleshoot. It seems there was a misconfiguration on the 4006 that caused it to spike the CPU to 100% every time we started moving IP phones over onto it. I had no idea that such a small detail could cause so much trouble but apparently it could. That one little mistake, when multiplied across 120 phones, really started to add up and it beat the snot out of the switch. Of course I didn't spot the problem right away, and as I've already mentioned it took me a good while to figure it out. Thankfully once I had it nailed and removed the erroneous configuration statements everything went smooth as silk. All the phones were moved over and we felt confident that everything was okay now. To prove it to ourselves we did a drop and reload on the switch for a trial run of the new configuration. Once it rebooted and linked into the backbone the IP phones all linked up to their Call Managers in under three minutes, a far cry better than the 30 or 60 minutes it was taking them to negotiate a connection earlier in the night with the broken configuration.

The downside to all this is that I have to be at work at 7:30 tomorrow, which means I'll be getting up in approximately five hours. Yuck.

Still I have a sense of accomplishment -- at least the migration is done and I was able to identify and resolve the configuration snafu that was causing all the trouble. Without my help my coworker would have once again rolled back the changes we had been attempting, and when all was said and done we'd be firmly wedged in the same place we started at. Knowing that I made that much of a difference and contributed to the success of the cutover makes it feel like the night wasn't a completely wasted effort.

Current Mood: satisfied
Current Music: Ra purring in my lap