This exact thing happened to realitychecknetwork hosting about 6 years ago (now rebranded to serverstack and digitalocean).<p>There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients.<p>"Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared.<p>They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean.<p>They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it.<p>Good cautionary tale for segregation of credentials and proper user key management.
This may be an unpopular opinion, but I want to preface this by saying: "before passing judgement, context is always necessary."<p>Mario Savio was a Free Speach Activist and organized a protest to protect the Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he says "there's a time when the operation of the machine becomes so odious... that you can't take part... and you've got to indicate to the people in charge that unless you're free, the machine will be prevented from running at all!" Applied to free speech, this notion of disrupting the functioning of an organization was lauded, because freedom of speech is just that important.<p>But let's shift to employment. Without employment, it's very hard to survive. And here's a situation where the people in charge has the upper hand in every arena- hiring, pay, work Place behavior... etc. How do we know that the ex-admin wasn't blackmailed by the CEO to come back to work for free to fix something, or future references will be negative? Why are we so quick to side with the employer in this matter when we know nothing of the situation at all? Why do we start calling the employee a felon? He hasn't even been charged yet.<p>My point is, context is important. Fine, corporations have the power to ruin your life as a deterrent to keep you from acting against their interests, and that's just the way society is. And fine, We're not all rational at every instance of life. The calculus of establishing status quo equilibrium of those two conditions/constraints is hard, but without context to the situation, who are we to decide who's right or wrong? Would you label Mario Savio wrong for protesting and urging protestors to prevent the operation of the college from functioning in the name of preserving Free speech wrong? No, because you've learned the context.
So in addition to the criminal side of things I guess the ex-admin wants to work in manual labor or fast food. There is no way in hell he'd have the references or pass the background.<p>BTW, we had a netadmin interview a few months ago. Guy was really smart, aced the technical and group interview. We were really looking forward to hiring him, and only needed to pass a background and reference check. HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.
Wayback Machine link if you want to know who they were:<p><a href="https://web.archive.org/web/20170603212121/https://verelox.com/" rel="nofollow">https://web.archive.org/web/20170603212121/https://verelox.c...</a>
This is so stupid. If you have a problem with your employer, either you quit or they fire, you move on, full stop. If you're in a relationship and someone isn't happy enough with you and breaks up with you, the dignified response is NOT to key their car. I see employment relationships mostly the same way. Either it works (for both) or doesn't (for either or both ends).<p>And having switched jobs quite a few times, the next one is always better for you, regardless.
Other than treating staff well, how would you go about stopping something like this?<p>As my own company is growing, we fully trust all employees, (limiting only what is essential), but, a dev ops guy if he was so inclined could technically do something like this... It always scares me.
Proper exit procedure should have disabled all access from this ex-admin..., unless s/he had some sort of cron job or launched some process that would execute commands at certain time? I am very curious to know how it was done.
If I ran a hosting company and all of my servers were compromised by ring -3 malware exploiting the Intel AMT vulnerability, the first thing I'd do is privately inform Intel that I intend to go public with the story and sue for damages, after which Intel would perhaps offer a very generous bribe for my silence and a week-long window to replace all of the server processors for free, on the one condition that I bury the truth by fabricating a story about an imaginary ex-employee who improbably was both smart enough to gain an administrative position in a large company while also being stupid enough to risk decades in prison for petty revenge over workplace drama.
Lots of comments are interpreting "ex-admin" as someone who was fired and <i>then after</i> went and did this. Just want to float the possibility that "ex-admin" could also mean someone was employed there, <i>then did this</i> and is now no longer employed as a result of doing this.<p>(Btw, IMO there is no excuse or justification for any admin or exadmin to ever do this. Among many other issues is the fact he deleted the data/work of individuals who had nothing to do with whatever "problem" he has with Verelox )
It's always interesting watching startups learn the lessons that thousands of enterprise learned along the way. "Why would you ever want offline tapes sitting in iron mountain, how inefficient".<p>Nothing is foolproof, but anytime you've got constant network access to every last copy of your data, you're begging to lose it. It's the reason why people who think one copy (redundantly dispersed or not) in AWS S3 is sufficient scares me to death. Is it unlikely Amazon would get hacked and have the entire thing blown up? Sure... but if we go to war with China I wouldn't want to bet my company on it.
Been waiting for a company to announce shutdown after this was posted: <a href="https://news.ycombinator.com/item?id=14476421" rel="nofollow">https://news.ycombinator.com/item?id=14476421</a><p>Possibly related?
A lot of 'managed' hosting providers are pretty bad with security, there still is a major provider that just gives root credentials to all servers to all techs not just admins, doesn't audit who accesses which credentials, and doesn't rotate credentials, doesn't rate limit dumping credentials...
That's before we go into more interesting issues with their security.
Frankly I am surprised this sort of thing doesn't happen more often ? In some ways it both restores some of my faith in people while reducing some of it at the same time in a different vector.
<a href="https://www.lowendtalk.com/discussion/116329/what-s-up-with-verelox-being-down" rel="nofollow">https://www.lowendtalk.com/discussion/116329/what-s-up-with-...</a><p>Some posts from Verelox staff towards bottom third of this forum page search for user name Verelox
This is why a hosting company needs to both segregate credentials to only what an employee needs for their job, as well as to revoke them the minute they leave the company.<p>Otherwise while the vast majority of your staff will be decent people and not cause problems like this, it just takes one angry ex staff member with a grudge to cause problems.<p>They also need to revise their backup system too. There should rarely if ever be a risk that any data is 'unrecoverable', yet their update says some data will just be impossible to get back.<p>As for the employee involved... well I hope they like the inevitable lawsuit their selfish, stupid actions will bring them. I don't care what you think of a company you worked for, there's no excuse to destroy their business through actions like this. Also, good luck getting any jobs in the industry after too. Because with this on your track record, no one will touch you with a ten foot bargepole.<p>So yeah, what a disaster all round.
Dick move from the ex-admin, but I'm curious to know what would compel an ex-employee to take such a brazenly criminal and traceable yet damaging action.<p>I'd like to know more, I think...