DigitalOcean lost our data and gave us $500

So if you were backing up your data to Tarsnap, then you&#x27;d be up and running as quickly as you could launch a new instance and redownload everything. And $500 credit is enough to power a micro droplet for 100 months, or a small droplet for 50 months. DO handled this well.<p><a href="http://www.tarsnap.com" rel="nofollow">http:&#x2F;&#x2F;www.tarsnap.com</a><p>EDIT: s&#x2F;years&#x2F;months&#x2F;g. Thanks.

So this is a technical problem I am having right now that&#x27;s preventing me from backing up a Postgres database completely (hope someone here can help).<p>I have a master Postgres database that is receiving a TON of transactions per second (I&#x27;m talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everything. Not only that but it impacted performance to the point where backing it up was just not feasible.<p>No problem.. just create a slave-DB and run pg_dump on that, right? We did just that, but the problem is that you can&#x27;t run long running queries on a hot standby (queries that take more than a minute).<p>What would you do in my scenario? With the hot standby, I technically am backing up my data, but I would have 100% piece of mind if I could daily backups in case someone accidentally ran a &quot;DROP DATABASE X&quot;, which would also delete the hot standby&#x2F;slave db as well.

The abrasive headline is kind of unfortunate, as the actual moral of the story given at the end is exactly the right takeaway: Never assume your hardware is infallible, so always have backups that you know you can use when your server experiences a wildly improbable catastrophe.<p>Also, very impressed by Digital Ocean&#x27;s response here. Given their reputation as a budget host, they really do put a lot of effort into service.

That&#x27;s <i>way</i> more compensation than I would have expected. AWS usually won&#x27;t even notify you until after the node has gone down.<p>Hardware failures happen; an application needs to be tolerant of it.

This is 2013. Why are we still talking about backups as a lesson learned? Is it because startups are skimping on Sys Admins?

It&#x27;s great you had backups, but why a write-up. Is it an attempt to smear DO&#x27;s otherwise good name? It&#x27;s an un-managed VPS so it&#x27;s your responsibility to keep backups of your box, not theirs. And hardware fails all the time, so you can expect this to happen anywhere.

&gt; And if you just launched and have a single instance running, let your alpha users know that there will probably be some downtime.<p>That&#x27;s true. But there&#x27;s no reason for extended downtime even if that instance goes down. Make sure your whole setup is described in chef&#x2F;puppet&#x2F;salt&#x2F;ansible&#x2F;cf&#x2F;whatever and even a rebuild from scratch takes only minutes then. There&#x27;s really little reason to skip that these days.

DO is affordable enough that the minimum you should run are 2 droplets. Having said that, I&#x27;m actually fairly impressed with the 500 credit and now you have no excuses to run 2 vms. Consider it a lesson learnt.

Alternate title: DigitalOcean went above and beyond their SLA for us.

DigitalOcean&#x27;s pricing page indicates that &quot;All cloud hosting plans include automated backups&quot;. (<a href="https://www.digitalocean.com/pricing" rel="nofollow">https:&#x2F;&#x2F;www.digitalocean.com&#x2F;pricing</a>) From the email you received, it sounds like this is clearly not the case. I wonder what other claims DigitalOcean is making that are not true.

That&#x27;s always a risk with servers. They can die and everything can do with them. But they had backups so they didn&#x27;t lose everything.

It seems that is very nice from DO. I would not expect them to be responsible of data loss in case of hardware failure.

This might sound a bit glib, but raid 5 shouldn&#x27;t really be used in modern storage.<p>If you ignore the performance issues (which can vary by device) its just not safe. Depending on the size of drive can take anywhere up 30hours+ to rebuild.<p>bear in mind that you tend to use disks that are all the same batch, it leaves you in the danger zone for far too long.<p>Your options are: somesort of clever RAID (ZFS type thing) Another type of clever RAID (Like the LSI chunk thingy in the DCS37000) RAID 10

Was this really a dual drive failure, or was this the rather common single drive failure plus undetected errors on a backup drive, that show up when trying to rebuild?<p>Because that happens a lot, and it&#x27;s very important to do a <i>full</i> read of every drive in the array at least weekly! You have two options for doing that:<p>If you are using linux md raid then run the &quot;check&quot; command, which automatically does the test using background I&#x2F;O (but does still impact things). On debian, and perhaps other distros too the mdadm command will do it every month by default. Make sure to set a minimum speed or it might never finish if you have a busy system.<p>You can also use the built in SMART on the disk to do a long self test. This also uses background I&#x2F;O and I think it has a bit less impact on existing operations. (But you have to have some idle time on the disk or it will never finish.) If you install smartmontools you can set smartd to do this test for you every week, and keep an eye on the results.<p>I personally do both, plus a short self test of the disk every night.

I truly believe that we did the best we could in this instance. Drive failures are always always unfortunate, even with backups, downtime exists.<p>That being said, we&#x27;re always genuinely looking to improve, and I&#x27;d welcome your feedback on how you feel we did and how you feel we could do better. Please do reach out to me personally john@do! Thanks. :)

&quot;we had backups&quot;.<p>Do you mean you had backups on digital ocean (using their backup service) or something else?

Good thing you had backups.<p>With that being said, these days it&#x27;s a good idea to use a deployment tool or configuration management system like puppet&#x2F;salt&#x2F;ansible&#x2F;chef&#x2F;etc, especially in a virtualized environment. This will help with scalability as well as situations such as these.

This is the reason why I moved all data away from my server instances. My images are hosted by cloudinary(with s3 bucket backup) and my databases are Amazon RDS instances. I don&#x27;t care if a server goes down, I can launch a new one in a matter of minutes (with ansible) without any data loss.

The author is sweet, his conclusion was &quot;always backup your data&quot; if it was me I would probably say &quot;I&#x27;m moving away, will never trust them again on my data&quot; ..

The $500 credit from DO is quite reassuring. Usually if the HD fails and your data is lost, your out of luck. I hear the &quot;horror&quot; stories of some hosts reusing consumer Hard Drivers between servers so learned, Your data is your responsibility. I&#x27;m glad the OP had backups but these failures happen, thankfully DO had the business sense to compensate them.<p>Seems good advertising for DO, as any knowledgable system admin knows Drives fail. DO could have not done anything.

Linkbait title, they handled it exceedingly well. Onus is on you to back up your data. You did not &#x27;lose&#x27; your data, given you had backups.

&gt; And if you just launched and have a single instance running, let your alpha users know that there will probably be some downtime.<p>How about instead &quot;alpha users should know that there will probably be some downtime&quot;. Multiple instances don&#x27;t really fix that.

Nice move from DO to give everyone $500 credit. As I remember, they don&#x27;t guarantee data safety (you still need backups even if they did). Double disk failure is a rare thing, but it happens.

Is DO apologizing here as a PR move, or do they make reliability claims that would lead you to think this sort of thing wouldn&#x27;t happen?

DigitalOcean proudly advertises that they use SSDs... A dual drive failure with data loss should be very rare. I wonder what happened.

So they now run raid5? I remember they boasted about raid10 a while ago, now they silently downgraded to raid5 :)

What is the best&#x2F;cost effectiv way to backup a windows server?

What are the best options for backing up DO externally?