vRRitti.com, providing sentiency

June 1, 2012

This morning a hacker was able to access a customer’s account on CloudFlare and change that customer’s DNS records. The attack was the result a compromise of Google’s account security procedures that allowed the hacker to eventually access to my CloudFlare.com email addresses, which runs on Google Apps. While we are still working with Google to investigate the details, we wanted to highlight it here to make people aware that they too may be vulnerable to similar attacks and provide a full accounting of what happened.

Hack a Long Time Coming

This attack appears to have begun in mid-May. It appears an account request was sent to Gmail for my personal email address. Google’s procedure asks for a number of questions to attempt to verify account ownership. We’re not clear on how the process works, but it appears that weeks after the process was initiated, the hacker somehow convinced Google’s account recovery systems to add a fraudulent recovery email address to my personal Gmail account. The password used on my personal Gmail account was 20+ characters long, highly random, and not used by me on any other services so it’s unlikely it was dictionary attacked or guessed.

Once the recovery email address was added, the hacker could then reinitiate the password recovery process and get reset instructions sent to the fraudulent email address. Those instructions were then used to reset my personal email this morning.

Google Apps and Privilege Escalation

Like thousands of other companies, CloudFlare uses Google Apps for email. When we first established CloudFlare.com’s email address, I listed my personal email address as a recovery email for my account. The hacker was able to use Google’s password recovery and have the password reset sent to my personal email for my CloudFlare.com address. Surprisingly, all CloudFlare.com accounts use two-factor authentication. We are still working with Google to understand how the hacker was able to reset the password without providing a valid two-factor authentication token.

Once the attacker had access to my CloudFlare.com email account, the hacker was able to access our Google Apps administrative panel. The hacker appears to have targeted a particular customer, and initiated a password reset request for the customer’s CloudFlare.com account. We sent a copy of these requests to an administrative email account for debugging purposes and, ironically, to watch for invalid password reset requests. The hacker was able to access this account in Google Apps and verify the password reset. At that point, the attacker was able to log into the customer’s CloudFlare account and change DNS settings to temporarily redirect the site.

Working With Google to Resolve

We were aware of the incident immediately. We have senior contacts at Google who we worked with in order to regain control of the Google Apps accounts (both my personal Gmail account and my CloudFlare.com account). We were able to revert the change to the customer’s account. We manually reviewed all other password reset requests and DNS changes. There were no other CloudFlare.com accounts that were accessed or altered.

To ensure that no other accounts can be compromised, we have invalidated all the password reset logs. We have also removed copies of password reset requests from being set to any administrative email accounts in case our Google Apps account is compromised in the future. From our investigations, it appears that at no time was our database accessed or any additional client data exposed. It appears this was, in effect, a very elaborate and sophisticated attack targeting one particular customer’s login information.

Protecting Yourself

My personal email address has been removed from any association with CloudFlare. I’ve also added two-factor authentication to my personal Gmail account — something that this incident highlights the importance of. I would recommend if you are using Gmail or Google apps, you take the following steps as soon as possible:

• Add two-factor authentication to your account by following the steps here;
• Ensure your password on your email account is extremely strong and not used on any other services; and
• Change any password recovery email to an account that you do not use for anything else and cannot easily be guessed by a determined hacker.

The final puzzle we don’t yet know the answer to is how the hacker was able to bypass Google’s two-factor authentication on CloudFlare.com email address. That is troubling. That should have prevented this attack, even if the attacker had the password, so it remains concerning to us that it did not. We are working with Google to understand how two-factor authentication was disabled. As we learn more, we’ll update this post.

Update (Saturday, June 2, 2012, 7:40 GMT): Just received notice from Google that they tracked down the issue core issue that allowed a compromise of the two-factor authentication system. Google reports that they discovered a “subtle flaw affecting not 2-step verification itself, but the account recovery flow for some accounts. We’ve now blocked that attack vector to prevent further abuse.” That’s great news. I want to reiterate that the Google Security team has, at all times throughout this incident, been responsive and attentive to the issue. In my opinion, they are the model of security on the Internet and we continue to trust them to power email for CloudFlare.com.

Update (Saturday, June 2, 2012, 19:37 GMT): We have found no evidence of unauthorized access to CloudFlare’s core systems or other customers accounts. We continue to work with Google to understand the nature of how the Google App’s platform was breached. In a review of the contents of the email accounts that were compromised, we discovered some customers’ API keys were present. In order to ensure they could not be used as an attack vector, we reset all customer API keys and disabled the process that would previously email them in certain cases to CloudFlare administrator accounts. If you’re using an app like the CloudFlare WordPress plugin, you’ll need to reenter your new API key.

We’ve received questions some questions from customers about credit card numbers. CloudFlare’s payment systems are designed to never see any credit card numbers. Credit card data is sent directly to a secure payment processor without ever passing through CloudFlare’s servers. This is designed to protect sensitive account information even in the case of a full breach by a fully privileged administrator.

More:
http://blog.cloudflare.com/post-mortem-todays-attack-apparent-google-app

Reader Comments:

• Google’s infra doesn’t seem to be any better or worse than running your own
• “Google’s procedure asks for a number of questions to attempt to verify account ownership. We’re not clear on how the process works, but it appears that weeks after the process was initiated, the hacker somehow convinced Google’s account recovery systems to add a fraudulent recovery email address to my personal Gmail account.” The real question is how he managed that (or did I miss something?). It seems like a big security flaw to me, one that would have prevented the entire situation. And why would he add a recovery email address? To do that you need access to the account, and if he had that he could already read your inbox and thus reset the cloudflare account. No need for any fraudulent addresses.
• It seems to me that there were two separate flaws in the logic here. One was within the Google Apps flow (which they have since corrected). The other seems to be in the decision to BCC the password reset link email to a compromisable email account. Lesson learned – you are only as secure as your weakest link.
• Somebody is not telling the truth here…the hacker claims that they were able to access cloudflare’s main server infrastructure and all customer informations but Matthew is claiming otherwise. For cloudflare to regain its reputation they need to come out with valid evidence that only CEO’s email was hacked and nothing else otherwise hackers win on this one.
• Its not a big deal to just bypass whole google’s reset password question and just change the password , It’s just a cookie work and everybody can do that , I used to do it every time I had problems with some accounts I usually hack :)