Let me give a little back story here: At around 10:45 CDT on Saturday, the 5 th I got a phone call from MegaPath Networks, the ISP that's servicing my DSL line. I was greeted by a technician who informed me that a machine on my netblock was involved in a few spam complaints. I got the necessary information from the tech and promised I'd take a look at it and find out what was going on. He told me that until I shot in an e-mail stating the issue was resolved that MegaPath had taken the step of blocking the IP of the offending machine to prevent any further spams from originating. This seemed like a reasonable thing to do on their part, so I wasn't going to complain. Plus it helped me stave off the immediacy of resolving the problem. So long as it was blocked I could get it looked into at my leisure. I wished the technician a good day and ended the call. When I sat back down at my PC I found that none of my Internet applications were working anymore. My browser couldn't establish a connection to any web sites out on the 'net, panther delinked from the IRC network, etcetera. What the hell? He said he only blocked the one IP address! So I started trying several of my other machines, and sure enough, none of them could get to out to the Internet either. My entire netblock was knocked offline! This wasn't what he told me would happen at all, I thought. Now my blood was starting to boil. I tried to call the TechSupport line, only to find I couldn't remember the number -- and I couldn't get it off their web site because my web browser couldn't connect to the Internet. It looked like I'd just been served a triple-thick milkshake of suck! A quick call to roho turned up the phone number I needed, and I dialed it up. Sure as anything as soon as I transferred to Tech Support I was dropped into a hold queue that just kept recycling over and over. I waited five minutes, then another five, and discovered that I was in for the Long Haul and nobody was going to get to me anytime soon.
Since I had access to the router I took advantage of the situation, telnetted in and started poking around. A brief review of the device's WAN Event History section showed that my system had dropped and re-established link with a message of "Profile Changed." Okay, that tells me whatever they'd done to institute this block had been done on my router and not upstream. That meant I had a chance at undoing it myself. I started poking around, since I wasn't sure exactly what they'd meant to do or had actually done. I poked here and there through the various settings, and finally found that my DSL line's "Connection Profile" had an IP filter associated with it. I pulled up the menu for IP filters and took a look -- sure enough, there was a rule dropping any traffic from the Internet to the offending IP. This seemed okay, so why was the rest of my network hosed up? I probed a little deeper, and then discovered where things had gone wrong. Rather than specifying a full 32-bit netmask for the IP that had gone rogue, the technician had accidentally put in my network's netmask. This had the effect of blocking all traffic from the Internet to my entire network instead of just the one machine he'd meant to quarantine. As soon as I figured that out I moved back to the connection profile, disassociated the IP filter from the profile and updated my settings. 27 minutes after I'd been taken off the network I was back online again, able to surf the 'net from my PC as well as my servers. I sat on hold for another 30 minutes, waiting to see if I could get a technician on the phone because I wanted to tell them what had happened. The longer I waited, the more pissed-off I was getting. It was becoming plainly evident that if I hadn't had access to my own router and had knowledge of how it worked (thanks to my job and natural curiosity) I'd still be offline, at the mercy of the call center as to when things would get fixed. When the impact of that hit me I really became white-hot, and the seventy-fifth repeat of the automated attendant saying "We're sorry for the delay, if you wish to leave a message, please press 1," pushed me right over the edge.
Let me make this perfectly clear. I wasn't upset that this technician had made a mistake -- that happens. Sometimes your brain has a bad neuron fire, or you just fat-finger something on the keyboard. Mistakes happen. My issue wasn't with the fact that a mistake had been made -- it was with the fact that, were I a normal, hapless person without access to that device that I'd have been completely screwed. I pay for business class service, which comes with a service-level agreement and everything. I certainly didn't think that I was receiving business-class service, being left high and dry for over 30 minutes like that with a network that was completely shut off from the rest of the world. That was, to me, the epitome of completely unacceptable service: to break the connection like that and then maroon me because they were understaffed. I pressed one so hard I figured I'd end up breaking the number pad on my poor cell phone. I walked through the various prompts, leaving my name, my number, my address and my account number. When I got to "Please briefly describe your problem," I went off. I tried to keep from yelling as I told them, "Yes, my problem is that this morning, at 1045 Central time, [name withheld] contacted me to say I had a machine that was spamming, and that he'd blocked it in the router. That's fine with me, but whatever he did blocked the rest of my network and left me hanging without IP connectivity for over 30 minutes. I sat on hold for 45 minutes and nobody answered my call. If I didn't know how to fix my router myself I'd still be waiting for somebody to pick up so this could get fixed. I think this is unacceptable. I'm very upset with this treatment, and first and foremost I want an apology. Second, I want to know how your technician screwed up my router and why he did it." Stab pound, so that the voicemail system records that portion of my voicemail, and then get prompted with, "Please leave your access hours or callback times." I thought I was hot before, after sitting forty-five minutes they're now asking when it'll be "acceptable for them to call me back?" How about instead of calling me back you people just took my call?!? When it prompted me for an answer I told them when they could call me back, all right. "Well, I just spent 45 minutes on hold and nobody was around to take my call. So since the number I gave you is my cell phone, and you've already shown no interest in servicing my problem quickly, you can call me back whenever you goddamn please. Call me back at Midnight, if it makes you happy. Because if I have to wait for you to call back to resolve this issue, then something is very wrong. You know what? From now on my recommendation as a customer to my friends and coworkers is to go to your competitor, Speakeasy. I don't need this hassle." And with that, the voicemail system was done. It wished me a pleasant day and ended the call.
I spent the rest of the day carrying my cell phone with me, waiting for them to call me back. No joy. Sunday didn't turn up any calls from them, either. This morning, however, when I got up and got ready to leave for work Lana told me that my cell phone had gotten a call at 6:30 (I got up around 6:45). I hadn't even heard it ring. I flipped it open and, sure enough, there was a missed call from a number that didn't show up on caller ID, along with a voicemail. "Screw it," I said. "They waited this long to call me back, it can wait until I get home to be listened to." When I got home from work today I pulled the phone off the charger, dialed up my voicemail and took a listen. It started off with the caller identifying himself: it was the same tech who called me on Saturday! His voice sounded really rough and uneven to me, almost as if he was trying not to cry or had been crying. As soon as he identified himself he told me he'd gotten my voicemail, and that he was very sorry, he hadn't meant to take me offline, that he wasn't sure why that had happened since he didn't do anything different from what he'd done to hundreds of routers before. He asked me again to accept his apology, and then said that even though I'd gotten it fixed myself that he would step me through what he'd done. These guys were taking what I'd said on the voicemail very seriously and quite literally. He did indeed step me through what he'd done to apply the filter, and reiterated that he was sorry that this had happened, and he wasn't sure why it had occurred, but that he was very apologetic that I felt MegaPath had disserviced me. Then he wished me a pleasant day and hung up.
I'm not certain exactly how to feel right now. Part of me says that I should feel good, because I obviously rattled the right cages up there and got their attention. Maybe it'll fix the problem, and I certainly wasn't wrong for giving them a piece of my mind when I was getting customer service that bad. I pay a lot of money every month to have the service I get, and it certainly isn't reasonable of them to take advantage of me like that.
On the other hand, I feel bad. The guy's voice was shaking and hoarse, as I mentioned. It sounded like he'd gotten the reaming of a lifetime. I feel bad if he got in trouble for what happened, because it wasn't his mistake that caused me to get upset... it was how the mistake was handled. Sitting for 45 minutes (Well, that's when I gave up and stopped counting because I left the VM. Who knows how long it would have been if I had stayed on hold?) waiting to get somebody on the phone to undo a simple typo is just poor customer service any way you look at it -- especially when you factor in how long it took for them to call me back! But that certainly isn't his fault, nor is it something he has control over or could fix. I don't even know if he got reprimanded -- maybe the dude just had too much coffee this morning. But no matter what, it wasn't my intention to make it seem like he was the sole cause of any problems. Maybe I'm just overreacting, letting my empathy get the better of me.
I just ... I dunno. Any way I look at it, this "victory" tastes of sour grapes.
Life is a highway