Sometime earlier this week, my personal server was compromised. For those who haven't been playing along at home, I have been doing hosting type things for a few years
I have done hosting, in various guises, for a few years now -- almost a decade.
I pay a monthly fee to rent a dedicated machine at a large hosting facility in Florida. I've been doing the "personal server" thing for a number of years, probably because I was a BBS SysOp before the Internet ate our scene and I moved down to Illinois. I used to run a SPARC box off a dedicated SLIP line via my 28.8k modem, which later evolved to hang off an ISDN line. When twanfox and I moved to the upstairs apartment at Autumn Chase we went through a few providers and finally settled with a 1.54Mbps SDSL service provided by MegaPath. I upgraded the machine from a SPARCStation 10 to a dual-processor Ultra Enterprise 2 and procured a giant APC Smart-UPS 2200 to keep the drives spinning and the router up it if the lights went out. When I moved out of the Hoffman Estates apartment in 2002 the machine was sent to live at a coworker's house so it could remain connected to the net on a nice, fast circuit with a dedicated IP address.
In 2003 I struck a deal with mindslide to share the cost of a dedicated machine at a facility she knew of in Florida. Things didn't go quite as planned and our partnership split up a few months after the original discussion, but I kept renting the server and migrated everything I had off the Ultra2 to this new box (this shuffle was further necessitated by the coworker's purchase of a house and resulting plan to drop the lease on the apartment where my machine was living). It wasn't running Solaris (still my OS of choice for servers) and it wasn't in the local area, but the added redundancy of multiple connections to the net, generator protection and an actual data center facility were hard to pass up when compared to running a machine in my bedroom off a single T1 and a moderately-sized UPS.
As I hinted above, the machine started out as a personal system, just a place where I could dink around with UNIX (which landed me my current career path) and run TinyFugue for chatting on the MU*. It was nice to have a machine I could do whatever I wanted with. If I wanted to install a software package, I didn't need to provide notice or get permission. If I blew the box up, nobody would care. Of course, this was lack of responsibility was not to last. Gradually my machine took on other tasks from what I had originally intended, like hosting e-mail for myself and my friends. Then I added DNS and a webserver and started hosting the web page for my domain. I think it was in 2000 or so I took on my first "customer," providing a home for FrostFire MUSH. Things just went downhill from there. I've been providing more and more shell accounts and then started providing for (gasp) money to interested friends and associates. It's gotten to be pretty full-service, with webmail, database back-ends for customer websites and the like. It keeps me busy and I don't make money off it (I'm not even "breaking even" on more than 1/4 of the expense of renting and maintaining the server) but I like doing it anyway. It's the geek in me -- I know I'm going to pay to have my little server-toy anyway, so other people might as well benefit from my insanity.
Which brings me back to this week's adventure.
On Wednesday morning I was sitting down at work, sipping on my coffee and preparing for a long day of listening to people building shit with their mouths instead of with their hands. As usual routine I logged into the server and started skimming the mail that had collected since I'd gone to bed. The subject of one in particular grabbed my attention: ** URGENT *** it said. Phishing attack on your server. I read on. The message warned that an attacker was using my server to harvest Amazon accounts and passwords. The informant included the phishing URL and so I was able to check things out and confirm -- sure enough, there was a very nasty set of web pages nestled into one of the subdirectories of a site I'd recently taken on hosting responsibilities for. I know the owner of the site and knew she hadn't done it -- especially since I hadn't yet given her access to the system to maintain her pages. I started getting a sense of low-grade concern, so I backed up the files for evidence and then wiped out the offending directory. Low-grade concern would later give way to frustration and outright dismay.
Okay, I've been compromised, I thought. And I don't know how bad the attack was. Did they r00t it? Are other bad things going on? I better go find out. Thus began a 6 hour journey through the box. I found a couple of things right away that made my stomach sink: programs like top suddenly refused to run, citing dynamic links that had NEVER existed on the server. I kept digging and I eventually got my confirmation: an IRC "bouncer" program had been installed on the machine, listening on an unauthorized TCP port, disguised to look like my production MySQL daemon. Several key binaries like /bin/su and /bin/login had been replaced with trojans. One tool reported a number of cloaked processes running and loadable kernel modules installed, along with evidence of signatures for two prevalent "root kits."
The first law of running a system is that when you've been compromised, you do not try to recover the system as it stands -- you can't be certain that you caught every little dastardly thing the attacker has done. The appropriate procedure is to back everything up for evidence if you can, back up your data files (so you can use them as incrementals to the weekly backups that you're doing... you are doing weekly backups, right?) and then burn the whole damn system to the ground so you can do an install from scratch with clean, trusted media. The problem with this is that the machine in the hosting facility doesn't have a tape drive, and I'm not physically there so I can't swap media in and do an install myself. This momentarily stumped me until I started just making tarballs of everything important (all 24 websites that I host, all the mail files for my users, all the home directories of my users, etc etc) and pulling them off the system to a temporary storage facility. I didn't want to give my attacker any indication that I'd noticed him so, aside from removing the page to prevent further phish from being hooked, I left most everything in place until I was ready to have the system nuked. Wednesday night I was up late, making tarballs and FTPing them down to my PC (hooray for 6Mbps DSL lines and FileZilla!). I stayed home from work on Thursday because I still felt relatively crappy and worn down (small wonder, right?). Time was passed copying more files around between machines and trying to make sure I'd covered all my bases by backing up every last configuration file that I might need again. At around 2:30 PM or so, while talking with shaddragon, I called it good and sent a service ticket to my hosting provider. In the letter I explained what had happened and gave their techs permission to burn the box down and do a complete reinstall. Shortly after the system suddenly went unreachable, which told me they'd yanked the machine from their network at the very least as a security precaution. All I could do was wait. Later that evening came a follow-up e-mail from the hosting provider -- my request to have Fedora Core 4 installed (since they won't do FreeBSD) would cost me $150/hr since they don't provision Core 4 at this time, just Core 3. I talked with points for a bit and he pointed me to an FC3->FC4 migration path that could be done remotely, which was perfect. I gave the representitive my blessing to do Core 3 and waited. And waited. And waited. Eventually I went to bed. Total cost of the day's effort? 19 some hours of my time, a bottle of vanilla Smirnoff vodka and three 2L bottles of Diet Sunkist. Oh yeah, and more hair from my head.
When I got up Friday morning I still wasn't feeling great, so I opted to stay home from work again. Good thing I did, because at around 10:30 in the morning I was contacted by the tech team to verify my request (I'd forgotten to give my authorization code) for the rebuild. Once they had the proper documentation they went to work, promising it would be done in "2 hours or so." I didn't get the system back until about 2:30 PM yesterday. I ran the migration and got the system up on an FC4 userland with an FC3 kernel. Since I wasn't about to go through all this suffering and NOT have the latest and greatest versions, I spent the following two hours and change wrestling with getting "yum" to work in a way that made sense to me. In the middle of the final bulk upgrade the system was suddenly halted by root, which severely pissed me off. Either I'd gotten hacked in the three hours since installation (mostly unlikely) or the hosting company had shut down my machine -- in the middle of a huge upgrade -- for no discernable reason. More waiting. Eventually the system came back and I went back to work. First I brought over the websites, then I had to screw around with the pre-installed Apache in order to make it run the way I wanted while supporting the features I needed. That was an adventure. As several of the websites I host depend on the DB back-end to make them go, I had to get the MySQL database system up and running next. This was a relatively painless install for me... but after all the work, I found Apache's PHP wouldn't talk with it. twanfox was able to lend a bit of insight into that and I got things straightened out. With PHP and MySQL talking I could finally do an upgrade on the phpBB system that FrostFire uses (just to make sure I was current... again, with this much pain I deserve the newest and best). With the websites in place I turned my attention to getting the first of the hosted MUSHes back online, just to make sure I could get predictable behavior on this new OS. I had one small issue and then FrostFire was up and rolling again, so I started working on e-mail. That was at around 8pm last night. I worked on e-mail until 4:23 this morning, at which point I gave up and went to bed.
My desire to do database-backed virtual mail accounts seems to have been a touch... optomistic. I'm in the process of giving it one last college try, then I'm going to fail back to the tried and true old way of just having a shell account for everyone who has e-mail on my system. Not my ideal way of handling it, but I can't have the mail system down much longer.
I've done things I know you'll never understand