I'm more worried about the following post in that thread:<p><begin quote>
--------------------------------------------------------<p>We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No biggie, we'll just update it, turn the server back on and be on our way.<p>Nope. We had to contact customer service to get back in to our account. After updating the card info we realize that all our droplets are gone. We reach out to customer service again in which case they let us know that when a CC expires, an automated process kicks off and deletes the droplets. We've been a customer for 2 years, surely they could pick up the phone and call us. We've spent thousands of dollars with them.<p>Plan B, let's restore them from the backups we've been paying for. Nope! When they delete your droplets, they also delete your backups.<p>This is where we ask to speak to them on the phone and are denied. I then ask if they are insane and why they would delete someone's servers and backups via an automated process without a human at least checking to see if it is a loyal multi year customer who has a simple lapse with their CC expiry.<p>We had to find a backup on a developers machine from over a month prior and rebuild data using various megtods that took close to a week. In the end, DI gave us a $500 credit.<p>You get what you pay for. Anyone who uses DI for production or anything more than a hobby app is playing with fire. They do not care, they are apathetic, they will screw you over and the throw you a credit for their terrible service as a half assed apology.
This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"
I'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.
If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it.<p>For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them.<p>On a production system, do not keep the database on the same server as the application.
Probably due to optimization of the conversion funnel, it's VERY easy to build things on DigitalOcean without understanding that - unlike many other *-as-a-service - droplets do not include backups and that is your responsibility. As a sysadmin, this is fine since I don't trust a single provider for both prod and backups anyhow, but I fear for those who are just getting started and don't realize this.<p>I agree that DO handled this situation appropriately, but they should warn people a little more deliberately when they sign up and create a droplet. A checkbox along the lines of "I, user, understand that I'm responsible for backing up my server", with a few links to some how-to articles of the excellent quality that DO is known for, would suffice and empower without reducing the conversion rate.
This is yet another reason to push forward the idea of static websites/webapps. There is no single argument to support the idea of using a database to power a 5-6 page website. Not even a blog with daily posts. How many sites out there have less than 10 pages, are updated maybe once per month, sport a contact form or newsletter subscription box? 90%? And how many of them use an insecure behemonth like WP or Joomla?<p>My new model: client wants a pretty responsive theme, I get the HTML version of it. Turn into a template I can use to spit out a static site (from my homebrewed CMS). I tell clients your monthly support charges are $0. If you ever need updates I charge hourly (most never call in months). Some clients want "a list of something" say products. If small and won't grow: a static csv file that also gets munched by the homebrewed CMS. Large list? A BAAS service with API.<p>I really don't care if the servers are gone/hacked/ransomware. Git clone, go.
I think Google Compute Engine does the right thing here: by default it uses "persistent disks" (network-attached redundant/highly-available block devices) for <i>all</i> disks. The only case I've heard of where persistent disk data was lost was a few acknowledged writes occurring just before an unusual lightning-induced power outage: <a href="https://status.cloud.google.com/incident/compute/15056" rel="nofollow">https://status.cloud.google.com/incident/compute/15056</a><p>For added protection, you can take regular snapshots. You only pay to store the diff from the last snapshot (so go ahead and snapshot often), and snapshot storage is geographically distributed.<p>(Note: I have no idea what EC2 does, maybe it's similar.)
I definitely wouldn't recommend DO's built in backup service in production based on my experience. It brings my site down every week since all I/O freezes during the backup. Sometimes the droplet doesn't recover and needs a hard reset. Apparently it's a known issue but hard to fix.
"Luckily, we made the decision at Spatie to host every site on it’s own droplet, so only one site was affected."<p>I think that's a poor lesson learned here. Were this me, I would have said:<p>"Luckily, all of our sites run on several servers, access data in a shared, replicated cluster, and a small shell script I wrote kept me from writing this entire blog post."<p>IaaS has only surfaced what has always been true: your data lives on little physical things that are screwed into a thing and goes through a controller that could fuck up due to cosmic rays.<p>Do better for your customers.
Two things to learn from this:<p>1. Never have a single point of failure. Relying on DO for Server+Backups is putting your eggs on one basket.<p>2. Your server state should be programmable. This is not quite easy for complex configurations. But today, DO has an API, we have Docker, and quite modern deployment tools.<p>Here is setup:<p>1. Github for the server state. Basically, a repository to configure and deploy my infrastructure.<p>2. Enable DO backups in case I mess up something and want a quick come-back.<p>3. File backups through Tarsnap. Since I use Docker, I have a volume container. Backup the volume container with Tarsnap.
This has happened to me with Amazon servers too, it's just the nature of the cloud. I think Digital Ocean handled this as well as can be expected.
That really stinks, but this should be a warning to you that in the future, you should have some kind of redundancy plan so data does not get lost. Even if Amazon decided to terminate all of my app instances right now, I would just need to rebuild them. No data is actually lost, and furthermore backups are made frequently so that if a server decides to explode one day, we're still covered. DigitalOcean droplets are really not meant to be holding your database and application server...you shouldn't have experienced "data loss" by losing a droplet because your database <i>should not have been hosted</i> on a droplet.
To my surprise, this HN thread has no link to any external backup solution guide or little-to-no suggestion of best way to backup your server to an external service or another VPS/backup server.
What if you need to scale up to more servers? What if your server gets hacked and you have to recreate it? What if you accidentally delete the server or perform an upgrade that completely breaks it?<p>None of these would be an issue if you've scripted server recreation from scratch and permanent data is stored in high availability shared databases and services like S3. If you're relying on weekly backups to save your server state and customer data you're just asking for trouble. I like that Heroku recreates your server once a day to force you to do this properly.
The weekly backup deal is something I really do not like with Digital Ocean, I think Linode does it better, I've been planning to move for a while now, I've just been super lazy, but this might be the push I need. 7 days of lost data is unacceptable to me, especially when I'm paying for backups.<p>> Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. A fourth backup slot is available for on-demand backups.
Although I have backups of my critical files, I don't want to try to rebuild from those files if I can help it.<p>After learning that Digital Ocean does no backups of their own (even for critical hardware failure on their own side) I've enabled weekly backups for an additional $1 per month.<p>Glad you posted this so that I know that option is available and really necessary. And who wouldn't pay $1?<p>Edit: It costs 20% but I only pay $5 for my personal server.
Does anyone have any good backup solutions to mitigate this? My agency uses a script we wrote in-house to more-or-less rsync our data to an AWS instance, but it's always seemed a poor way to handle it. I'm using DO's backup service for my personal site which was always supposed to be a temporary solution (the timing of the backups is inconvenient as mentioned by the article).<p>A better solution would be wonderful.
The comment thread on this post makes me very happy. Having recently switched all my production instances to individual docker containers, I'm very happy to hear that the consensus seems to be that server instances should be killed and spun back up like bacteria. There's hope for humanity yet.
I do understand the part where they terminated and killed of the VMs. I do not, at all, understand why they deleted the backups. How can the CEO sit in his chair and make that decision? If he didn't and he doesn't know, why is he still in that chair?<p>Even damn Netflix preserves data for 10 months. Deleting backups is making 100% sure that the customer doesn't come back to you. It's amazingly stupid.<p>Unless you're as smart and as big as Google, use the damn phone. Don't use the word <i>automated</i> at all. It's not automated if one fool made a decision and another one wrote the script.
Oh well, It's not like DigitalOcean doesn't tell you to enable backups. And you should not expect that hardware failures are now a thing of the past just because it's in "the cloud".
If you're paying for backups and they get deleted the moment there's an issue with payment, ur doin' it rong. I don't care whatever the TOS says, they're being completely shitty to customers. Sure there should be regular off-site backups. No one is arguing against that. But the fastest restore is usually from the closest source, and that's partly why you'd pay them for it.
I hadn't heard of the backup providers mentioned in the article. Does anyone have experience with those or with other recommended solutions?<p>I can re-implement the infrastructure of my VMs fairly easily (thank you Ansible), but backing up content outside of the provider's built-in options is something I haven't played with yet, and would obviously be the crucial piece.
Tell me if I'm doing something wrong: my two EC2 servers both use EBS including for the root device, so I'm not using any device-attached storage. If Amazon "lost" my instance, it would appear to me to be just a reboot.<p>And yes, I do a nightly backup of databases, webroot, /etc and some other directories.
I use DO as a test bed for a lot of stuff. I ended up destroying/recreating so many droplets I just started using Chef. Also, I come from a major background in AWS, ChaosMonkey in prod, and a very strict push-button + nuke & pave deployment strategy.<p>So, all this strikes me as "lol you didn't know this?"
Whats the point of raid if raid card failure resulted in complete loss of data? I came across an article a week or two ago that discussed various alternatives, I think ZFS with checksums, which unlike raid will not replicate corrupt data from drive nearing failure to the healthy drive.
Holy hell, obviously where critical data is concerned, redundancy is something you should insist on, but this definitely sucks. Seems ridiculous that all you got from DigitalOcean is a $15 credit though. I'd have expected something like 6 months of paid backups comped.
We are also experiencing a huge problem with performance of DO servers as we scale. Especially databases. With stories like that , a decision of moving to AWS seems pretty obvious.
Anyone have any experiences with running VM's with ceph storage? I've used it for other things, but I know that VM hosting is quite popular. Care to share any stories?
I had two DO machines with a unrepairable file system after a crash and reboot. Shit happens. Never rely on a _any_ VPS. I treat them as throw-away boxes that can fail anytime.
No surprise for me. It also happened on DO for me one time. It seemed I've had my machine on a wrong rack/server there. At the end you get what you pay for I would say... you can have luck on your DO server but there is a reason why Rackspace and Amazon still exist nowadays! If you can afford one of the bigger ones go for it.
you should hear what people say when we suggest that they make their own backups in addition to the ones we provide for them.<p>adults acting like children. nothing more, nothing less.<p>take responsibility for your own data. back it up yourself.
I don't know how to write this without upsetting quite a lot of people, but RAID!???!<p>I run Redundant disk controllers and hard drives on ZFS. And it's cheaper than a decent RAID system.<p>First thing I do with a new server is disabling RAID.