Feren (feren) wrote,

  • Mood:

Let me put it this way...

A sort of skeleton timeline is best for this:
  • It took me two hours to drive into work today
  • Today is Tuesday, so I had that damnable weekly staff meeting this morning
  • One hour into the staff meeting, the entire network here at the home office goes away
  • One hour after that, our troubleshooting has produced nothing, so we pull every single uplink GBIC in our 6509 core fiber router
  • This fixes the problem... but....
  • This leaves us with the problem every floor in the tower we call "home" is now isolated from the rest of the LAN, and all the servers are isolated from the users
  • Plugging the GBICs back in one at a time does not isolate the problem. Instead, the problem just magics itself back into existance after they've all been reactivated
  • One hour after the failed attempt to isolate the piece of equipment that's to blame for this, we decide that the core itself is to blame and reboot it
  • Our problem on the admin VLAN goes away, equally magically
  • Our CEO and our CIO want to know what the problem was, how we fixed it, and how to prevent it from happening again. They find little humor in our answers of "Something, dunno, can't tell ya."

  • Subscribe

    • Post a new comment


      default userpic

      Your IP address will be recorded 

      When you submit the form an invisible reCAPTCHA check will be performed.
      You must follow the Privacy Policy and Google Terms of use.