Sunday July 12, 2009

Presumed Guilty: The Problem with Google Blacklist Removal (And The Solution)

2 Comments

Written by Raina Gustafson

Topics: google, hackers, malware, security

Are you here looking for help with your blacklist dilemma?

If so, I invite you to skip the article and request help here. If you’re a geek like me interested in all the juicy, glorious details, by all means read on.

Google Blacklist Removal: How We Got a Google Ban Lifted

It was definitely a Trojan Horse… and our defenses were nearly as weak as ancient Troy. Excited and encouraged by recent business developments—reaping the benefit of referral relationships we’d cultivated—we were eager and willing to inherit a site created by two prior development teams. The client’s requests were straightforward, and fairly typical. They’d grown weary of working with developers far away, and wanted a local technical partner to take stock of the site’s gender-bending past (from Ruby to the PHP framework Symfony), fix a few bugs, and be ready to oversee a rebranding campaign when the time came.

Everything seemed fairly innocuous. Sure, it was a drag to sift through the vestigial Ruby code that previous developers had left in place, and the homepage redirect was seriously ill-advised, but nothing hinted of what was to come.

 

Banned From Google

A couple weeks into the project, everything temporarily went pear-shaped. I’d seen the Google malware warning screen a time or two, but never on a site that I cared enough about to seriously question what was going on. When I suddenly found myself needing to care, the cryptic quality of Google’s ‘Safe Browsing diagnostic page’ weren’t particularly helpful. I happened to be the roommate of a sysadmin at the time—who was a former (by choice) Google employee—but he was nearly as equally mystified as I was.

Nevertheless, we followed Google’s ‘reconsideration request’ guidelines, but to no avail since we hadn’t yet identified what the problem was. After researching Stop Badware, I was convinced that the forced redirect put in place by the previous developers was a major red flag. We removed the redirect, and again submitted a reconsideration request. Still, no dice.

Having tried the least invasive approaches while still not thoroughly familiar with either the code base we’d inherited nor the server/hosting company that the site resided on, I decided to stop messing around. Nothing less than control over every single variable was going to satisfy me. This was my ‘take no prisoners’ approach:

1. Transfer all work to my own servers and point the DNS to it, thereby isolating the domain away from the flagged network IP block
2. Set up a completely static home page and submit a reconsideration request with the home page and only the home page present
3. Swore on our mothers’ graves to the client that we hadn’t done anything to create this situation, and that we truly were not at fault. We also ensured them that they could rely on us to do everything in our power to make the best of a bad situation. To be honest, I’m not sure they believed us. But if I’d been in their shoes, I’d have been freaked out too, and probably wouldn’t have known who to believe. (Though we now know when it happened, we still don’t know who or what created the security breach.)

This approach—though rather drastic—was bullet-proof, and worked. We got the Google ban lifted. Of course, we still needed to re-instate the rest of the code base and maintain our clear status. Here’s how that went down:

1. We moved the Symfony files from the old server to the new one (initially a Media Temple Gridserver) via scp
2. While sorting through the files manually, we noticed an index.html file in the root web directory that didn’t serve any purpose (due to Symfony’s architecture, the index file that gets served to the browser resides elsewhere). Upon further examination, we saw that the file was full of links. At that point we knew the security of the site had been compromised. We later found evidence to prove our own innocence.
3. Meanwhile, we learned the hard way that the (gs) doesn’t play nice with Symfony, so we moved again to a Dedicated Virtual.
4. We still hadn’t identified exactly where in the code base the breach was. Though we’d sanitized many pieces of code, problems still seemed to be systemic. When trying to redeploy the site using native Symfony tools like fix-perms, the files seemed to spontaneously reset their ownership and permissions. This was infuriating. We finally strong-armed Symfony (and updated to the latest stable version of the 1.0 branch since the since the code we’d inherited was missing about 18 months of important security patches and enhancements), and re-released the site hoping for the best.

Serendipity Emerges

Around this same time, Eric asked for a little help customizing a Blogger theme for a client of his called Dasient. As they emerged from stealth mode before my very eyes, I realized that they were the answer to my Web Anti-Malware prayers. I immediately employed their free diagnostic tools, and received extremely valuable insight as to what was happening with our client’s site.

Simultaneously, Google had blacklisted the site again. The sinking feeling of disappointment was tremendous. But Ryan, who did not know about Dasient yet, had been doing his own research, and with the aid of Unmask Parasites and Firebug had ascertained that the site continued to establish a connection with a known malware site every time he loaded a page. Yet searching through the site for the unwelcome URL was useless. The hackers were far too sophisticated—with intricate knowledge of the Symfony framework and the specific exploits they could tap—to make eradicating their handiwork that easy. This was a multi-pronged attack, and the hackers had their fingers in the site through PHP, Javacript, regex and more. Once we painstakingly identified each component, we were able to grep through the site and remove all traces of the errant code. Equilibrium established. Finally. And we were vindicated when we discovered in the server log files that the initial security breach had taken place back in February—months before we’d even met the client.

In a subsequent conversation with Neil Daswani, a co-founder of Dasient and author of Foundations of Security: What Every Programmer Needs to Know, I learned how and why these types of attacks are increasing in frequency while malware distribution through email is declining. We further discussed the solutions that Dasient is providing the web community. In particular, they offer a quarantining service in the form of an Apache module. In short, it gets installed on the web server, monitors for attacks, and if it detects any malware it will sanitize the html before being rendered in a browser. I think this is amazing, and am thrilled to be working with them as a field beta tester.

For more free information on what you need to know about web security, visit Dasient.com to download white papers and more.

• Home
• About
• Design
• Development
• Marketing
• Contact
• Blog
• Clients

Email:

All the Queen's Men LLC
548 Market Street ECM #57766
San Francisco, CA 94104
Phone & Fax: 1.888.850.2006