How We Got Owned by a Few Teenagers (and Why It Will Never Happen Again)

I feel really bad for the phpfog guys. But given the situation, I think they handled it admirably well - kudos to them. No software is secure and this could have happened to anyone. Especially startups who have to take shortcuts at the very beginning.<p>I know the attackers were just kids but I have to admit pursuing legal action sounds very tempting - even to just act as a deterrent to others. If they had just put up phpfogsucks.com, it <i>might</i> have been ok. But tweeting trash from their twitter account, redirecting their root domain to phpfogsucks, etc - are all not cool at all and should have <i>some</i> consequences.

The blog post is riddled with the words "luck" and "timing" which brings doubt into my mind that the team can actually take full responsibility for their actions.<p>"aware of the potential security threat " but they left it for the next week, who honestly here would do that?<p>I have also seen comments around the web of migrating to Php Fog because of how they handled the situation. If you are one of these people please enlighten my mind as to how you came to such a logical decision or how much you get paid per year.<p>Also if Php Fog could enlighten us on how their terms of agreement will work in the case where our intellectual property is stolen on no fault of our own.<p>Save your sympathy for the sites that are still down, four days and counting

I am bothered by some of the language in this post:<p>- <i>we were aware of the potential security threat behind post-deploy hooks and were about to disable them [...] but...</i><p>- <i>we were days away from replacing this server</i><p>- <i>They were a short-term stopgap measure we had been planning to replace</i><p>To me, it sounds like the real problem could have been stated as "We were lax on security," but almost worse than that is the lack of accountability that I sense from company. Yeah, maybe it won't happen again, but it's hard to be full of confidence to buy into a service like that.

"We have hired professional white hat hackers with government level security experience to attempt regular pen tests on our system..."<p>I guess whenever I read this kind of statement from now on I'll be thinking of HBGary and chuckling a bit inside.

I mentioned this last time, but I don't think anyone was interested, but the "John" guy is compwhizii (same handle on Twitter) who runs the forums (facepunch.com) for garrysmod, a very popular game. I will be curious to see how garry (owner person) responds to this, or if he already has.<p>Elliot is apparently VERY scared and blames John (compwhizii) (edit: not john, he blames someone else called supersnail1): <a href="http://www.facepunch.com/threads/1071855-A-member-of-Facepunch-may-cause-me-to-be-sued" rel="nofollow">http://www.facepunch.com/threads/1071855-A-member-of-Facepun...</a><p>Here is (compwhizii) Johns reply: <a href="http://www.facepunch.com/threads/1071855-A-member-of-Facepunch-may-cause-me-to-be-sued?p=28754506&#38;highlight=#post28754506" rel="nofollow">http://www.facepunch.com/threads/1071855-A-member-of-Facepun...</a>

The phpfog guys really deserve praise for being so open on this issue. As a fellow engineer, being able to learn from their mistakes and see exactly what they could have done <i>ahead of time</i> to avoid the disaster is priceless.<p>Just goes to show that those with the time to spend are the most likely to break your stuff, even if you pay "professional white hat hackers" to test your system.

It seems like incredible coincidence that allowed this to happen but when I think back to all of the security incidents I've been involved in, it always seems this way.<p>I guess the best way to think of it is that badness on the internet is like water. It will flow into every tiny crack in your wall you haven't sealed up tight. A crack in a dam doesn't leak less because its in an "obscure" location.

Goes to show you why the DRY principle (I might be stretching that analogy here, but bear with me) is important here - if you have old stuff lying around in production that was cloned a long time ago, you might forget about it and open yourself up to unfortunate incidents like this.<p>PHP Fog is doing great work to make the PHP ecosystem easier to work with, and I hope they didn't suffer too much from this mistake.

While it is admirable and good that they have learned from their mistakes and are taking steps to reduce the likelihood of getting hacked in future, to say "never again" is to paint a big red bullseye on yourself.

Wait...their model is an EC2 instance per customer? The normal limits Amazon imposes are 20 reserved or on-demand instances and 100 spot instances per region. You can request more, but will Amazon really accommodate a one instance per customer model?

Leaving the doors to your house wide open does not grant every passerby the right to enter.<p>So, yeah, PHPFog screwed up and did that. Then these kids went in, threw paint on the walls, smashed some windows, etc.<p>PHPFog was stupid - they admitted that.<p>The kids were criminal.<p>The first is not illegal - the second is.

What a crazy story. If the timelines are accurate there was an extremely small chance of this happening. Bad luck all around.<p>My site is still down, guess i'm in the unlucky 1%.

Ugh, you shouldn't try writing an apology after not sleeping for days. Sleep on it first, always sleep on it. Talking about prosecution and explaining this with a framing that it was all a fluke caused by the only person who was silly enough to IM you with a confession... add one more person who will never be a customer of yours with an apology like that. Now I know you're irresponsible.<p>Seriously don't write official blog posts for your company while you're experiencing "I was just in the field for days trying to fix this stuff" emotions.<p>Calm down, then try and be graceful about the fact that you were hacked by a few clueless kids. (Clueful kids don't let you know who they are.) Then try and figure out how to protect yourself against people with a clue.

Wow, that is quite the list of security measures that they had almost but not completely/correctly implemented, or hadn't got around to yet.<p>I guess the real moral of the story is to finish what you begin, or don't keep putting security off until it is convenient for you.

Never? I would be cautious about issuing a challenge like that.

Congratulations to PHPFog. They've managed to direct the attention to the 16 year old kids rather than their own incompetence.<p>Is it me or no one mentions the lack of expertise of the PHPFog team in PHP and Systems Administrations.<p>Sure kids broke in and the way they published their findings was despicable. The fact remains that PHPFog was utterly broken to pieces and the exact essence of the problem is simply the lack of knowledge in their field.<p>I am very disappointed by the tone of the blog post and think PHPFog don't really have a notion of what they are doing. I would much rather seem them where they belong, in the Ruby world where their experience is.

Their response and abilty to turn the situation around is a case study in dealing with a difficult situation. Kudos! I'm saving their response and will use it when dealing with things. Being able to have a counter party to identify has definitely helped in handling the situation. I didn't realize how powerful that can be until I saw this, I learnt something new.<p>Its a brilliant piece and a great start/way to restore faith and recover from what must be a pretty grueling ordeal. Good job.

Great to see disclosure. This can happen to anyone, and more so for startups, where labor is short, focus is on developing features. Using the phrase "Never Happen Again" is a bit strong though. Security is risk management; spend until you can accept the remaining risk while still maintaining profit and avoid being a hacker's low-hanging fruit.

This post convinced me <i>not</i> to use PHPFog. They reveal more in their lack of foresight and security prevention measures than their response to what was otherwise a fairly trivial exploit. I am not sure this blog post was helpful in convincing customers like me that want to feel that their infrastructure providers are on top of things.

Here's an interesting tweet from one of their developers.<p><a href="http://twitter.com/ReinH/status/50348989366796288" rel="nofollow">http://twitter.com/ReinH/status/50348989366796288</a><p>&#62; Your password in the database is SHA512 encrypted, but we're not taking chances.<p>I hope he knows what he's talking about and is just tired from the past few days.

I remain in two minds about idea of charging the kids.<p>There is no doubt they did some things they should not have. And I don't doubt there can be a decent case built against them. But as someone who actually had something from his teen years come to bite years later, it's not pleasant. At least in my case it was a MAJOR maturing moment(also the worst day of my life). May be it will take a lawsuit to get these kids to mature up...to that extent anything that gets em to mature up before they <i>really</i> get screwed would be fair.<p>I'm not merely advocating another chance but actually something that gets these kids to be a tad more thoughtful about their actions. It's not always easy to do that when you are 16 and full of adrenaline.

I am sure, many of the HN users here would have found at least a loophole in similar systems in the course of time. What I do in such situation is letting the service know about the flaw. Isn't that the ideal behaviour ?

&#62; Eliminate shared hosting failover server – We may never do shared hosting failover again if we can not guarantee its security. We might do a non-realtime failover to automatically launch a new instance for you, but this experience taught us what a bad idea this can be.<p>What does <i>realtime</i> mean in this case? Anyway, this isn't the only option. They could keep a few bare instances of their php stack online and simply run the deploy script instead of the image creation script. That ought to be able to run in under ten seconds I think.

This feels like a business model where the lean/MVP approach isn't quite appropriate. A lot of things fall out of that decision, not the least of which is that the exposure surface area you get from an environment that allows user-sourced code on purpose is enormous. I feel for the guys going through this but there were a lot of errors in the wild all at once to allow this to happen.

There is no such thing as bad publicity! Kudo's for turning lemons into a viral blog post! Although, if I understand correctly, you were reusing passwords and storing them in plain text! This is an ABC123 computer security nono. Thank goodness it was just some young script kiddies and not someone with malicious intent!

great to hear all the details so quickly so that others building similar systems aren't in the same situation. as fellow PHP'ers its also great to hear that you are not blaming it on PHP somehow (no fuel for the php haters).

<p><pre><code> 2:56:45 AM Elliot : then I used the method detailed by turby 2:56:46 AM Elliot : to gain root </code></pre> Has anything been said about what this method was?

So, shouldn't the first thing you learn as a hacker include how to mask your physical location so as not to have the FBI knocking on your door?

I feel for the people at phpfog.com, but this is a bigger blow to cloud computing.<p>Customers who are already pretty risk averse to their data being stored in the cloud would see this as another reason not to take the risk.<p>The cloud computing consortium needs to work on a stable stack as well as figure out how to audit that it works properly.<p>In addition, it calls for security ahead of features. Given that phpfog is funded, they'll need to implement the equivalent of a bleeding edge stack and a locked down stack.

php... a language by amateurs, for amateurs. phpfog... a service by amateurs, for amateurs.