Yep, there's nothing quite like these 12:00 AM upgrade windows. When I should be sleeping I'm working on Cisco kit and dealing with frustrating bugs that keep me from getting the right level of IOS onto the gear. It's amusing to note that these annoying bugs come to me after a weekend of frustration and stress from dealing with equipment for the upcoming con.

The fun started after the upgrade was done and the router came back online with the upgraded IOS version. I noticed the box was acting pretty choppy and while a small amount of suck can be expected after bringing up your BGP session and slurping down the entire Internet table, this was far beyond the level of pain I was prepared for. Diagnostics were performed and that's when the real fun began, because it was at that point at which I got to see innarestin' error messages like these in the system logs:

Nov 13 00:14:10.680 CDT: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x606F4898 0x615CE9A0 0x615D8788 0x615D8EB0 0x615E19D8 0x615EFE08 0x60A26810 0x60A26E18 0x60A27900 0x609CD384
.Nov 13 00:14:17.812 CDT: %FIB-3-NOMEM: Malloc Failure, disabling CEF -Traceback= 0x606F4898 0x615CE9A0 0x615D8788 0x615D8EB0 0x615E19D8 0x615EFE08 0x60A26810 0x60A26E18 0x60A27900 0x609CD384
.Nov 13 00:14:32.636 CDT: %SYS-2-MALLOCFAIL: Memory allocation of 4084 bytes failed from 0x60DE41EC, alignment 0
Pool: Processor Free: 25104 Cause: Memory fragmentation
Alternate Pool: None Free: 0 Cause: No Alternate pool
-Process= "BGP Router", ipl= 0, pid= 172 -Traceback= 0x606F4898 0x60851118 0x60857BE4 0x60DE41F4 0x60DE6274

What makes me bitter and angry about this outcome is that the box in question is $EMPLOYER's primary Internet edge router: a 7206VXR chassis populated with various PA's and an NPE-400. I shouldn't see errors about RAM but wait... how delightful! This box only has 256MB of RAM! Whereas the identically-configured standby router (that is almost never used) has the NPE's maximum of 512MB. This means my primary router can run BGP, or it can run CEF, but it can't run both -- but the unused standby can. That's splendid and is completely indicative of a well thought-out architecture for redundancy. And by that previous statement I mean is not at all any of those things.

Ra's perched on my desk and looking at me as if to say, "Dad, go to bed so I can curl up with you and be warm!" So I guess that's what I'll do. First thing I'll have to do when I get into the office tomorrow today is go make a fuss at my boss and see if we can buy a RAM upgrade. Hey, 512MB sticks for the NPE400 are only $185 on eBay. Our budget is, as always, very tight... but I'm sure we can swing that somehow if we engage in some creative accounting.

How’d I get so white and nerdy

Tags: hate, tech, work
Current Location: home
Current Mood: tired
Current Music: PC fan humming