Like most not-quite-major-but-certainly-not-minor changes we would have gotten done about two hours earlier if there hadn't been a sneaky problem that required some time to troubleshoot. It seems there was a misconfiguration on the 4006 that caused it to spike the CPU to 100% every time we started moving IP phones over onto it. I had no idea that such a small detail could cause so much trouble but apparently it could. That one little mistake, when multiplied across 120 phones, really started to add up and it beat the snot out of the switch. Of course I didn't spot the problem right away, and as I've already mentioned it took me a good while to figure it out. Thankfully once I had it nailed and removed the erroneous configuration statements everything went smooth as silk. All the phones were moved over and we felt confident that everything was okay now. To prove it to ourselves we did a drop and reload on the switch for a trial run of the new configuration. Once it rebooted and linked into the backbone the IP phones all linked up to their Call Managers in under three minutes, a far cry better than the 30 or 60 minutes it was taking them to negotiate a connection earlier in the night with the broken configuration.
The downside to all this is that I have to be at work at 7:30 tomorrow, which means I'll be getting up in approximately five hours. Yuck.
Still I have a sense of accomplishment -- at least the migration is done and I was able to identify and resolve the configuration snafu that was causing all the trouble. Without my help my coworker would have once again rolled back the changes we had been attempting, and when all was said and done we'd be firmly wedged in the same place we started at. Knowing that I made that much of a difference and contributed to the success of the cutover makes it feel like the night wasn't a completely wasted effort.
