September 25th, 2001


Let me put it this way...

A sort of skeleton timeline is best for this:
  • It took me two hours to drive into work today
  • Today is Tuesday, so I had that damnable weekly staff meeting this morning
  • One hour into the staff meeting, the entire network here at the home office goes away
  • One hour after that, our troubleshooting has produced nothing, so we pull every single uplink GBIC in our 6509 core fiber router
  • This fixes the problem... but....
  • This leaves us with the problem every floor in the tower we call "home" is now isolated from the rest of the LAN, and all the servers are isolated from the users
  • Plugging the GBICs back in one at a time does not isolate the problem. Instead, the problem just magics itself back into existance after they've all been reactivated
  • One hour after the failed attempt to isolate the piece of equipment that's to blame for this, we decide that the core itself is to blame and reboot it
  • Our problem on the admin VLAN goes away, equally magically
  • Our CEO and our CIO want to know what the problem was, how we fixed it, and how to prevent it from happening again. They find little humor in our answers of "Something, dunno, can't tell ya."

    • Current Mood
      confused confused

    Working late...

    I have to make up the hour and a half that I was late today thanks to traffic, so I am working late. The even better part is that we've still got network problems, so I may be staying even later.

    Bleah, bleah, bleah.
    • Current Mood
      annoyed annoyed