Log in

No account? Create an account
entries friends calendar profile Feren's dART gallery Previous Previous Next Next
Hackers can blow me. - Paint It Black
Living the American dream one heartbreaking piece at a time
Hackers can blow me.
This is a belated entry. It was intentionally delayed to try and maintain an advantage in a game of cat-and-mouse I was playing. Since the Super Happy Mega Fun Round is over now and I've returned to the usal day-in day-out dance, I can post this.

Sometime earlier this week, my personal server was compromised. For those who haven't been playing along at home, I have been doing hosting type things for a few years

I have done hosting, in various guises, for a few years now -- almost a decade.

I pay a monthly fee to rent a dedicated machine at a large hosting facility in Florida. I've been doing the "personal server" thing for a number of years, probably because I was a BBS SysOp before the Internet ate our scene and I moved down to Illinois. I used to run a SPARC box off a dedicated SLIP line via my 28.8k modem, which later evolved to hang off an ISDN line. When twanfox and I moved to the upstairs apartment at Autumn Chase we went through a few providers and finally settled with a 1.54Mbps SDSL service provided by MegaPath. I upgraded the machine from a SPARCStation 10 to a dual-processor Ultra Enterprise 2 and procured a giant APC Smart-UPS 2200 to keep the drives spinning and the router up it if the lights went out. When I moved out of the Hoffman Estates apartment in 2002 the machine was sent to live at a coworker's house so it could remain connected to the net on a nice, fast circuit with a dedicated IP address.

In 2003 I struck a deal with mindslide to share the cost of a dedicated machine at a facility she knew of in Florida. Things didn't go quite as planned and our partnership split up a few months after the original discussion, but I kept renting the server and migrated everything I had off the Ultra2 to this new box (this shuffle was further necessitated by the coworker's purchase of a house and resulting plan to drop the lease on the apartment where my machine was living). It wasn't running Solaris (still my OS of choice for servers) and it wasn't in the local area, but the added redundancy of multiple connections to the net, generator protection and an actual data center facility were hard to pass up when compared to running a machine in my bedroom off a single T1 and a moderately-sized UPS.

As I hinted above, the machine started out as a personal system, just a place where I could dink around with UNIX (which landed me my current career path) and run TinyFugue for chatting on the MU*. It was nice to have a machine I could do whatever I wanted with. If I wanted to install a software package, I didn't need to provide notice or get permission. If I blew the box up, nobody would care. Of course, this was lack of responsibility was not to last. Gradually my machine took on other tasks from what I had originally intended, like hosting e-mail for myself and my friends. Then I added DNS and a webserver and started hosting the web page for my domain. I think it was in 2000 or so I took on my first "customer," providing a home for FrostFire MUSH. Things just went downhill from there. I've been providing more and more shell accounts and then started providing for (gasp) money to interested friends and associates. It's gotten to be pretty full-service, with webmail, database back-ends for customer websites and the like. It keeps me busy and I don't make money off it (I'm not even "breaking even" on more than 1/4 of the expense of renting and maintaining the server) but I like doing it anyway. It's the geek in me -- I know I'm going to pay to have my little server-toy anyway, so other people might as well benefit from my insanity.

Which brings me back to this week's adventure.

On Wednesday morning I was sitting down at work, sipping on my coffee and preparing for a long day of listening to people building shit with their mouths instead of with their hands. As usual routine I logged into the server and started skimming the mail that had collected since I'd gone to bed. The subject of one in particular grabbed my attention: ** URGENT *** it said. Phishing attack on your server. I read on. The message warned that an attacker was using my server to harvest Amazon accounts and passwords. The informant included the phishing URL and so I was able to check things out and confirm -- sure enough, there was a very nasty set of web pages nestled into one of the subdirectories of a site I'd recently taken on hosting responsibilities for. I know the owner of the site and knew she hadn't done it -- especially since I hadn't yet given her access to the system to maintain her pages. I started getting a sense of low-grade concern, so I backed up the files for evidence and then wiped out the offending directory. Low-grade concern would later give way to frustration and outright dismay.

Okay, I've been compromised, I thought. And I don't know how bad the attack was. Did they r00t it? Are other bad things going on? I better go find out. Thus began a 6 hour journey through the box. I found a couple of things right away that made my stomach sink: programs like top suddenly refused to run, citing dynamic links that had NEVER existed on the server. I kept digging and I eventually got my confirmation: an IRC "bouncer" program had been installed on the machine, listening on an unauthorized TCP port, disguised to look like my production MySQL daemon. Several key binaries like /bin/su and /bin/login had been replaced with trojans. One tool reported a number of cloaked processes running and loadable kernel modules installed, along with evidence of signatures for two prevalent "root kits."

The first law of running a system is that when you've been compromised, you do not try to recover the system as it stands -- you can't be certain that you caught every little dastardly thing the attacker has done. The appropriate procedure is to back everything up for evidence if you can, back up your data files (so you can use them as incrementals to the weekly backups that you're doing... you are doing weekly backups, right?) and then burn the whole damn system to the ground so you can do an install from scratch with clean, trusted media. The problem with this is that the machine in the hosting facility doesn't have a tape drive, and I'm not physically there so I can't swap media in and do an install myself. This momentarily stumped me until I started just making tarballs of everything important (all 24 websites that I host, all the mail files for my users, all the home directories of my users, etc etc) and pulling them off the system to a temporary storage facility. I didn't want to give my attacker any indication that I'd noticed him so, aside from removing the page to prevent further phish from being hooked, I left most everything in place until I was ready to have the system nuked. Wednesday night I was up late, making tarballs and FTPing them down to my PC (hooray for 6Mbps DSL lines and FileZilla!). I stayed home from work on Thursday because I still felt relatively crappy and worn down (small wonder, right?). Time was passed copying more files around between machines and trying to make sure I'd covered all my bases by backing up every last configuration file that I might need again. At around 2:30 PM or so, while talking with shaddragon, I called it good and sent a service ticket to my hosting provider. In the letter I explained what had happened and gave their techs permission to burn the box down and do a complete reinstall. Shortly after the system suddenly went unreachable, which told me they'd yanked the machine from their network at the very least as a security precaution. All I could do was wait. Later that evening came a follow-up e-mail from the hosting provider -- my request to have Fedora Core 4 installed (since they won't do FreeBSD) would cost me $150/hr since they don't provision Core 4 at this time, just Core 3. I talked with points for a bit and he pointed me to an FC3->FC4 migration path that could be done remotely, which was perfect. I gave the representitive my blessing to do Core 3 and waited. And waited. And waited. Eventually I went to bed. Total cost of the day's effort? 19 some hours of my time, a bottle of vanilla Smirnoff vodka and three 2L bottles of Diet Sunkist. Oh yeah, and more hair from my head.

When I got up Friday morning I still wasn't feeling great, so I opted to stay home from work again. Good thing I did, because at around 10:30 in the morning I was contacted by the tech team to verify my request (I'd forgotten to give my authorization code) for the rebuild. Once they had the proper documentation they went to work, promising it would be done in "2 hours or so." I didn't get the system back until about 2:30 PM yesterday. I ran the migration and got the system up on an FC4 userland with an FC3 kernel. Since I wasn't about to go through all this suffering and NOT have the latest and greatest versions, I spent the following two hours and change wrestling with getting "yum" to work in a way that made sense to me. In the middle of the final bulk upgrade the system was suddenly halted by root, which severely pissed me off. Either I'd gotten hacked in the three hours since installation (mostly unlikely) or the hosting company had shut down my machine -- in the middle of a huge upgrade -- for no discernable reason. More waiting. Eventually the system came back and I went back to work. First I brought over the websites, then I had to screw around with the pre-installed Apache in order to make it run the way I wanted while supporting the features I needed. That was an adventure. As several of the websites I host depend on the DB back-end to make them go, I had to get the MySQL database system up and running next. This was a relatively painless install for me... but after all the work, I found Apache's PHP wouldn't talk with it. twanfox was able to lend a bit of insight into that and I got things straightened out. With PHP and MySQL talking I could finally do an upgrade on the phpBB system that FrostFire uses (just to make sure I was current... again, with this much pain I deserve the newest and best). With the websites in place I turned my attention to getting the first of the hosted MUSHes back online, just to make sure I could get predictable behavior on this new OS. I had one small issue and then FrostFire was up and rolling again, so I started working on e-mail. That was at around 8pm last night. I worked on e-mail until 4:23 this morning, at which point I gave up and went to bed.

My desire to do database-backed virtual mail accounts seems to have been a touch... optomistic. I'm in the process of giving it one last college try, then I'm going to fail back to the tried and true old way of just having a shell account for everyone who has e-mail on my system. Not my ideal way of handling it, but I can't have the mail system down much longer.

I've done things I know you'll never understand

Tags: , ,
Current Mood: busy busy
Current Music: Assemblage 23 - Skyquake

21 thoughts or Leave a thought
spoothbrush From: spoothbrush Date: November 5th, 2005 06:30 pm (UTC) (Link)
Yuck yuck YUCK. I hate that whole mess. Luckily I haven't had to deal with anything like that in a long time, probably because I don't really muck around with servers any more, but it makes me feel so... violated.

Which, when you think about it, is actually the case.
linnaeus From: linnaeus Date: November 5th, 2005 06:32 pm (UTC) (Link)
Perhaps I'm a lousy geek, but the information in this post that I expect to be of the most practical use to me is: Vanilla Smirnoff + Diet Sunkist.

And for that alone, I thank you.
linnaeus From: linnaeus Date: November 5th, 2005 06:35 pm (UTC) (Link)
Oh, and curb smiles are definitely required for the fuckers responsible for this bullshit.
enveri From: enveri Date: November 5th, 2005 08:07 pm (UTC) (Link)
It works with any vanilla liquor and orange-flavoured soda.

Vodka and diet are just our preferences. :)
linnaeus From: linnaeus Date: November 5th, 2005 09:06 pm (UTC) (Link)
Vanilla vodka sounds good to me because I have it, and it's less likely to bring other dissonant flavors to the party (like vanilla flavored... gin?).

Diet orange soda sounds good because any flavor problems I have with aspartame tend to get masked by the alcohol. And non-diet orange soda is a little much for me anyway.

I fully intend to give this a try in relatively short order. :) Anyway, sorry I missed you guys today. Is there anything you need or are waiting for from me conwise? If so, shoot me an email or call...
enveri From: enveri Date: November 5th, 2005 09:08 pm (UTC) (Link)
As long as the scantily clad women are in my room, I can't think of anything!

You'll be at the meeting tomorrow, I trust?
linnaeus From: linnaeus Date: November 5th, 2005 10:15 pm (UTC) (Link)

Scantily clad women?
enveri From: enveri Date: November 5th, 2005 10:16 pm (UTC) (Link)
Don't tell me you didn't get that request??

I wanted a bunch of scantily clad women waiting in my room when I got there.

Dammit! I can never get good help!
linnaeus From: linnaeus Date: November 5th, 2005 10:24 pm (UTC) (Link)
Ah. Well, I'll work on that. :)
moryssa From: moryssa Date: November 6th, 2005 04:44 am (UTC) (Link)
Can I volunteer?
duckhunter From: duckhunter Date: November 5th, 2005 11:10 pm (UTC) (Link)
Now you know why I do site-admin the way I do. Basically, because my geekhood sucks. :)

Although I've been lucky - after we got the DSL straightened out (which included an accusation from my provider that I had moved my appartment building 5000 feet farther from the CO without telling them), we've only had two major bungings - One while I was on vacation a couple weeks back.

Let's just say that as much as I suck as a site-admin, the assistant site-admin (roommate) was far worse. :)

Anyway, I feel for you, and I'll wave my normal bending/breaking fees if you find the nutless bastage that did this. Although you will have to pay for travel. :)
ronbar From: ronbar Date: November 6th, 2005 12:03 am (UTC) (Link)
Sorry you got rooted, but at least it was a kiddie. Such a pain in the ass to get all that re-installed, especially if it was built up slowly over the years and you don't remember all the details and gotchas from the first time around. Or if you're lazy like me and move PITA-to-install dependency-hell applications and services between machines using scp instead of finding the latest versions of everything and reinstalling from scratch.

Crap like this is why I no longer run anything useful at home anymore, aside from SSH so I can get in remotely. My email accounts end in yahoo.com and google.com these days.

I could probably build my own car to drive to work and learn a lot doing it, but buying something used at Carmax gives me a lot more free time to goof off on the internet.

Did you have any auditing turned on before the compromise? Can you run the things that need to run as root in jails?
rustitobuck From: rustitobuck Date: November 6th, 2005 06:39 am (UTC) (Link)
Ow. That really bites.

Actually, I have recovered a machine from being rooted, without burning the machine. It has since been moved to new hardware and fresh OS, but I used RPM's checksum database to find the compromised programs.

That's when we turned off FTP for good. FTP was part of how the hacker got in...got into a user account and used an exploit to elevate privilege.

And started running yum update every once in a while to get the recent patches.

I actually back up machines over the net now, pretty much everything but /usr, store the data here at home. Automatically. Cron scripts grabbing tarballs via SSH, or rsyncing database dump directories.

I used to have a really elegant script that made an exclude file for tar that excluded all RPM files that hadn't changed from the package install and weren't marked config files. Since one of the things in the backup is the list of RPM packages installed, I could make sure the packages are there, then do a restore.

I usually seem to end up building mod_php from a source RPM for some strange reason or another.

And PostfixWiki looks cool; I've got to do some Postfix upgrades to improve the accuracy of my spam detection in the face of thousands of spams a day. I don't have the nerve to hook a database up to the email system. Not yet, at least.
chebutykin From: chebutykin Date: November 6th, 2005 03:18 pm (UTC) (Link)
I owe you drunkening. Big time.
From: hysd Date: November 6th, 2005 05:17 pm (UTC) (Link)
Man, that really blows. :( Was really quite worried about what had happened to panther when I couldn't access it! Meh. Stupid crackers and their rootkits. Even more sad that you have to Fedora instead of something else. Boo!

Guess I should also contact you in some form for a password reset. Thanks so much for the time and effort you put into this Rolly. You're a gem.
From: onceagainplease Date: November 7th, 2005 08:18 pm (UTC) (Link)
hey man, i am still being denied access to my webmail and i cant seem to connect via scp, were passwords changed?
feren From: feren Date: November 11th, 2005 11:35 pm (UTC) (Link)
Yeah, now that accounts are back up and running I need to set your password to something new. Old passwords are, as far as I'm concerned, compromised and useless.

Catch me on IM and I'll get you squared off.
tuftears From: tuftears Date: November 7th, 2005 08:23 pm (UTC) (Link)
Ow! Sorry to hear you had the 'interesting' time of it, man. Did you ever figure out what happened to cause the reboot in the middle of the update?
mindslide From: mindslide Date: November 7th, 2005 09:02 pm (UTC) (Link)
Stan pointed this entry out to me. I didn't read the whole thing (just the first part...I've got to read about this type of shit in tickets at work all the time, you know what I mean?). He told me that in an AIM convo with him, you said that you had to have the hosting company restore a backup of the site. In the parts I read, you said something like the hosting company doesn't have tape backups. So I don't know which to believe.

The hosting company DOES have tape backups. I've seen their TSM backup system...like every time I've been down to Florida / hung out with friends there. They had a smaller TSM backup thinger when I worked NOC there, and when I worked tech support right before that, the shared hosting folks constantly asked for restores (which we would do and charge them for), so I can't imagine their low-level colo boxes being different.

So I'm not sure if you got or are planning to get a restore like that, or if Stan misunderstood wtf you were talking about, or what, but if you would like me to ask my friends there to drop billing for whatever the shit you need, just let me know.
feren From: feren Date: November 11th, 2005 11:39 pm (UTC) (Link)
[I'm not sure if you got or are planning to get a restore like that, or if Stan misunderstood ]

Something got lost in the translation between what I said and what got relayed to you via Sancho (and I generalized a bit above for the sake of brevity) but that's neither here nor there as I've been able to bootstrap the websites and user accounts from backups I make personally.

I appreciate the offer to get billing for a restore eliminated, tho. :)
From: (Anonymous) Date: November 9th, 2005 06:41 am (UTC) (Link)


Very soon someone will come up with a way to stuff Ipods up the holes of those who destroy servers. Read your post, couldn't understand much. Joe
21 thoughts or Leave a thought