How Hackers Took Over My Site, What I Did About It, And How You Can Stay Safe

This is a recap of our session from Drupalcon 2015 in Los Angeles.

About 6 months ago, the Drupal security team announced one of the worst bugs in the history of Drupal. The recovery period has been very challenging for our community, though I think it's safe to say now that the storm has passed.

One of my personal sites, http://whaleocalypse.com/, was affected by the attack. I've been able to verify that the attack I experienced was the same attack that an overwhelming majority of Drupal sites experienced. What follows is a comprehensive post-mortem on that attack, including how it was done, and how you can ward off similar attacks.

A couple of disclaimers before I begin. First, I'm going to confess to some very insecure practices on whaleocalypse.com. Understand that whaleocalypse is a play thing, I knowingly took huge risks with that site because I had nothing at stake if it was lost. Quite honestly, I was pretty excited to learn that this disposable site been hacked because it gave me the opportunity to learn more about security. I would never take these same risks with my professional client work.

Second, and perhaps more importantly, I'm going to reveal how this hack was pulled off in very, very specific detail, including a large amount of fully functional exploit code. Checkout the code on Github. If at any point you feel yourself going "Hey, you're telling people how to hack my site!" I'd invite you to stamp that feeling down. If you are still vulnerable to any of the techniques described here, I'm sorry to say, you were already hacked weeks ago.

Table of Contents

How This Hack Works
Video Explaination #1

The problem lies in database.inc, specifically in the function expandArguments(). Note lines 739, 745, and 755 in that file. It might be a little hard to follow, but the gist is that Drupal trusts the value of $i on 739 without sanitizing it. This is a problem because, in the case of the user login form, $i is generated based on the name attribute of the html <input> element.

Which means this hack just so, so, so easy to pull off:

Using this vector, I trivially worked up a little drupalgeddon client that will allow me to inject complex SQL statements into any vulnerable site.

Here it is in action:

Checkout Stephan Horst's examples to get a broader sense of what's possible with this bug.

How This Hack Works in the Wild

It's all well and good to talk about the theoretical danger presented by this bug. What I want to know is: how was the bug actually used in the real world by the real hackers who broke into my site. In order to answer this question, I spent a great deal of time examining the exploit files left on my server, and reverse engineered the tools that would have been used to create them. Check it out on Github.

I want to note here that, according to data released by Acquia 68% of hacking attempts in the first week of druppalgeddon were of the variety I'm going to describe here. The second most common attack style—adding a new admin user—accounted for 26% of all attacks. The user add style of attack is well understood, so I won't go over it here.

The attack that I, and most drupal site owners, experienced I'll call the "menu_router" attack. It works as follows:

1. Create a new entry in the menu_router table whose "access_callback" is file_put_contents and whose "access_arguments" are a filepath and the contents of a file. This creates a new page in drupal to the effect of http://example.com/backdoor, where "backdoor" is a random string.
2. Visit the new exploit page http://example.com/backdoor. When Drupal checks to see if the user should have access to the page, it will run file_put_contents('path/to/backdoor.php', '<?php //contents of attack file here ?>'). This creates a backdoor file which allows the attacker to execute any PHP on your server.
3. Using the newly created backdoor.php, run some code which will make a request to another server and download a secondary attack file. This secondary attack file is a file upload form. It gives the attacker the ability to upload any file to your server.

Video Explaination #2

Also, here's another video overviewing the whole exploit kit, and the defense kit.

Video table of contents:

• Intro 0:01
• The Anatomy of a Hack 0:55
• Step 1: Inject the SQL 4:17
• Step 2: Create a Backdoor to run PHP code 10:54
• Step 3: Create a Backdoor to Upload Files 13:14
• Defense procedure 1: prevent SQL injection 22:21
• Defense procedure 2: prevent malicious file uploads 29:01
• Defense procedure 3: prevent arbitrary code execution 34:15

Step 1: Inject the SQL into the menu_router table

Using some strategy akin to the SQL injection tool I showed above, the hacker would have inserted a new row into the menu_router table. That row looked like this:

How to Defend Against This: Drop All POST Traffic in Varnish

If you haven't yet, UPGRADE TO DRUPAL 7.32 OR HIGHER. If you are reading this, and you are running a Drupal site on version 7.31 or lower, your site has already been hacked, probably several times by many different individuals.

That said, this is a good opportunity to harden your site against all SQL injection. If your site has a handful of logged in users, and serves mostly anonymous traffic, one precaution you can take is simply rejecting all POST data in Varnish—almost all SQL injection vectors will make use of POST traffic. You can do this trivially in your VCL file like so:

Site administrators can still login by visiting Apache directly on its new port, which is now 8000. You'll prevent non-administrators from visiting Apache directly by using an IP access control list in your Drupal .htaccess file. To accomplish this, simply prepend the following to your htaccess file:

Now, when a hacker tries to POST to your site they'll get an error:

See: Defense Procedure 1 at 22:20 in my video above.

Step 2: Create a backdoor that allows you to run any PHP

Now that we've created our menu_router entry, it's time to visit our new page. When we do so, a new file will be created.

As soon as the attacker visits this page, assuming your file system is writable, the following file will be created:

I spent a long, long time staring at this file during my post mortem. It was finally this comment on drupal.org that cracked the case for me. Can you tell what it does? Spoiler alert! It's a backdoor that allows the attacker to run any PHP code. Once this file is installed on your server, all the attacker has to do to execute PHP is send an HTTP request like so:

GET /modules/poll/zkwv.php HTTP/1.1
Host: exploited.com
Cookie: Kcqf3=base64_decode; Kcqf2=["preg_replace", base64 encoded]; Kcqf1=[any PHP, base64 encoded]

I've reversed-engineered a deobfuscated version of it, to make the inner logic visible:

How to Defend Against This: Set Proper File Permissions

Its worth noting here, that the most common attack seen during Drupalgeddon would have failed on any site that had set proper Unix file permissions. If you haven't already, you can do so automatically with this script. See Defense Procedure 2 at 29:00 in my video above.

Step 3: Create a backdoor that allows you to upload files

Now the hacker has the ability to run any PHP on your server. And of course there's a great deal of damage that is possible with that power, but I've been able to determine what this hacker actually chose to do with this power. In addition to the arbitrary code execution backdoor I demonstrated above, I found several attempted backdoors. The most telling was this one, which I found inside the locale module:

Note, that is not my 404 page, that is the attacker's 404 page. Which means my server tried to execute some code intended to download a file. Which also means that we can go and see what file the attacker was trying to download:

I recognize this file! All told, I found four copies of this file on my server, and two copies of 404 pages in which the attacker attempted to download this file. When you compile it into HTML it's an uploader page, which will allow the hacker to upload any file to your server:

So not only do I now know what code this attacker executed and what it was for, I know the name of the town in Romania where he or she is from

How to Defend Against This: Upgrade to PHP 5.5.0

Remember, this is the second of two backdoors the attacker is going to install, and it is wholly dependant on the success of backdoor number 1 (which gave the attacker arbitrary code execution). Backdoor #1 is wholly reliant upon a weakness in preg_replace which allowed you to evaluate the haystack as PHP. This weakness was removed in PHP 5.5.0.

So while the attacker might still get backdoor #1 installed, they won't be able to use it, and they will not be able to create backdoor #2.

Putting it All Together

Considered together, you come up with a script like this. This is my best guess at the script this hacker actually used to attack my site. Once a hacker has executed this script against your site, he or she will be able to inject any SQL into your database, run any PHP on your server, and upload any file to your server.

Why create all these backdoors?

OK, if I may speculate wildly for a moment: this attacker was not interested in changing any content on my site. They didn't deface the homepage, or use my server for spam as far as I can tell. All they did was install obscure backdoors with random file names. So if I had to guess, the goal of this particular attack was to compile a large database of backdoored sites, and then sell the database. So maybe eventually this could have been used to steal data, but that doesn't seem to have been the direct goal of this particular attacker.

Was This Really Drupalgeddon?

In a word, no. A two words, not hardly. In a gif:

Let's just review some facts:

• Anyone hosted on Acquia, Pantheon, Blackmesh, or Commerce Guys Platform.sh was safe, whether or not they patched.
• Anyone who's file system was unwritable to Apache was safe from the most common attack.
• Anyone running php 5.5.0+ was safe from the most common attack.

The discussion of this bug has so far centered largely on it's potential for damage. And, fine, I admit in theory this could have been really bad. But here in the world where we actually live, it's just impossible not to notice—nothing bad really happened. I've read lots of cases of hackers creating fraudulent users and uploading backdoor files, but I have yet to find—despite considerable effort—a single person blogging about a `DELETE FROM node` attack. As I learn more and more about this bug, I am increasingly persuaded that such an attack did not occur.

As Stéphane Corlosquet wrote on the acquia blog, across tens of thousands of (failed) hacking attempts, none were destructive, or even visible to the end user:

We could not find any query intended to change the content or destroy sites: attackers were only interested in installing backdoors to take over the site or server at a later point in time, and make the intrusion unnoticeable.

So just a casual suggestion among friends, but maybe we should stop analogizing this bug with the literal end of humanity and start regarding it as what it was: another day in the life of a sysadmin.

How I Recovered

TL;DR I restored from backups. Blair Wadman gives a pretty good overview of what it takes to secure your server, but briefly I:

• Used the drupalgeddon tool to check for exploit files (it found all of them, by the way).
• Copied my whole codebase from the server down onto my desktop and diffed them against the last known good version in git (this is how I know the drupalgeddon tool worked).
• After 360 days and 5 hours of continuous uptime, I re-installed Linux from scratch. It's a good thing too, because I've been meaning to patch the shellshock vulnerabilty (lol).
• Deployed my local backups to the new server.

In Conclusion

I just want to close by saying that I think this security event could have been a lot worse. The big Drupal hosting providers were patched before the vulnerability even became public, so very few high value Drupal targets were hit. Of the Drupal sites that did get hit, the most common attack seems to have been pretty innocuous. This is a case where open-source worked more or less as it's supposed to; more eyeballs on the problem brought an incredibly obscure bug to light, and let us fix it—whereas a closed source product would have kept chugging along with the vulnerability unpatched. While this has not shaken my faith in Drupal as a whole, it has made me sit up and take security a little more seriously.

References

DrupalBest PracticesDrupal How toDrupal PlanetLearning SeriesTechnology ToolsPost tags: DrupalgeddonDrupalDrupalconDrupal Secuirty