Why does Google allow the hacker and the account owner to keep resetting passwords in rapid succession? The timeline indicates that two lengthy ping-pong sessions took place during the incident. That kind of behavior should immediately raise a red flag. How often do legitimate users reset passwords alternately from two different locations 10 times in 15 minutes?<p>I'm surprised that Google doesn't detect two people fighting for control of one account. They could have easily detected ping-pong sessions and and locked both parties out of their accounts for a couple of hours. Or they could have penalized the newly added recovery address by forcing an exponential delay between resets using that address. This is not the first time I've heard of somebody breaking into a Gmail account while the account owner is using that very same account.
Rule #1 when publicizing security incidents: always publish something else to the blog within 1.5 hours so that the security incident isn't the top post.<p>Edit: semi-tongue-in-cheek per comments below; as a CloudFlare customer I went to the blog when this first came up expecting to see something but bounced when the first post was a discussion of SSL BEAST since that was the hotness back in the fall of 2011.<p>I do believe it was not planned, but I also feel that vulnerability disclosures should be pinned for a while somehow if possible. I think one way this is done is having a separation between 'new feature' blog and 'ops' blog.
Part of this attack bears a certain resemblance to the recent Bitcoinica compromise, in that, from what I understand, they were also forwarding admin emails to personal accounts, one of which was compromised leading to the attacker gaining control of the bitcoinica virtual servers. Cloudflare were fortunate - at least the changes made were reversible, whereas the bitconica compromise resulted in the virtual servers being unrecoverably deleted after breach and theft.<p>There are lessons to be learned from both incidents.
Does anyone else get the feeling that the attacker is going to be someone the Cloudflare team knows? Firstly they would have had to have known Matthew's phone number. Then, assuming the attacker always had the plan of disrupting the target site, they would have had to have known that the password reset mails were BCC'd to admins.
O.o This reminds me so much of the hack sequence from the game Uplink. The game was based on "hacking" but intentionally used hacking techniques from Hollywood :) It was pretty fun. Anyway for the highest-level targets, you had to get a voiceprint from the phone of an admin, crack the password on the box, and break the encryption while bypassing monitors.<p>Just found out: Uplink is on Steam, and in the Ubuntu Software Center now.
The hack seems very well planned - I wonder just how many smaller sites have been hacked the exact same way as practice, and not picked up since they didn't have direct access to Google's security people?
Pretty intense hack... I think Google gives you the option of setting up a recovery phone to receive voice or SMS messages. It looks like SMS may be more secure.<p>It also sounds like he didn't have 2 factor setup on his personal gmail account. I wonder if that would of helped.
So, the "Five Whys" analysis came up one short, eh?<p>Just kidding, this is a great level of detail and much appreciated, to understand CloudFlare's process and how to protect against or recognize these tactics elsewhere.
Is flaw # 5 (or #1 depending on how you look at it) not having two-factor auth on the personal account? Or does account recovery by-pass two-factor auth by design?
The funny part - cloudflare still hosts the DNS for the guys who claim responsibility for this attack:<p><pre><code> Domain Name: UGNAZI.COM
Registrar: ENOM, INC.
Whois Server: whois.enom.com
Referral URL: http://www.enom.com
Name Server: LEE.NS.CLOUDFLARE.COM
Name Server: RUTH.NS.CLOUDFLARE.COM
Status: clientTransferProhibited
Updated Date: 29-may-2012
Creation Date: 22-jan-2012
Expiration Date: 22-jan-2013</code></pre>
Sounds like if the hacker had just done it out of hours, perhaps when the person in question was asleep, they would have had uncontested access to the accounts and the hack might have been far more damaging.
I'm amazed Cloudflare got a response from Google that quickly. I'm a paying Apps customer and I don't see responses for 24 to 48 hours on security incidents, not to mention that Google doesn't have a stellar reputation when it comes to things like "support".<p>Goes to show, it's always who you know (or it's bullshit, which is less likely). Or I don't have enough users.