Hard to believe we are 129 comments into this and there are zero mentions of 'borg' in the comment threads ...<p>For those that don't know, borg is a backup utility[1] that has been called the "holy grail of backups"[2].<p>It takes your plaintext files and directories, chops them into gpg-encrypted chunks with encrypted, random filenames, and will upload (and maintain) them, with an efficient, changes-only update, to any SFTP/SSH capable server.<p>My understanding is that the reason people are using borg instead of duplicity is that duplicity forces you to re-upload your entire backup set every month or two or three, depending on how often you update ... and borg just lets you keep updating the remote copy forever.<p>[1] <a href="http://borgbackup.readthedocs.io/en/stable/" rel="nofollow">http://borgbackup.readthedocs.io/en/stable/</a><p>[2] <a href="https://www.stavros.io/posts/holy-grail-backups/" rel="nofollow">https://www.stavros.io/posts/holy-grail-backups/</a>
The best solution I found as an alternative to an off-site copy is to set up a USB drive and something like a Raspberry Pi in my car. When I pull into the garage, the house server senses it and syncs new data to it.<p>This can be supplemented further by having it auto sync to a computer in your office whenever you pull into work. Add in some monitoring so you can get a phone alert if any of the synced copies are more than a few days out of date, and you are almost golden.
There really isn't a story here, Kev. You blew a power supply on your Synology. You even said it yourself: "maybe this is a problem with the enclosures and the disks are fine".<p>If you'd put your disks into a replacement Synology unit, you would have been back online - config, data, and all - within a few minutes.
Many years ago I watched someone stuff the DVDRW (remember them?) which contained all his stuff into a work PC’s pioneer slot loader drive to get some music off it. We stood there and the drive went bzzt, clang, bzzt, clang then sped up way faster than it was supposed to go. This was followed by a large bang and bits of DVDRW coming flying out and then a crunching noise.<p>From this I learned about single points of failure.<p>Edit: also in the decade and a half since I learned that you should never trust a magic box, magic piece of software or magic container file system for backups. A plain file system you can just copy your shit back from is the closest thing to a guarantee. Also it’s cheaper to curate your data carefully than end up with 4TiB of crap you’re too scared to deal with on your hands.
Kudos to Synology for having a process to recover on a Linux box if the Synology box craps the bed. I was reading this story thinking "this is a good lesson on why you don't buy an expensive SAN, unless you can afford to buy two".<p>But, the other lesson is: Backups. Sounds like backups were shut off 6 months prior.<p>The other lesson is: Monitoring. Backups were going to the USB drive, but it that stopped working at some point. Unless you have some tested monitoring of your backups, you are likely to lose data.<p>Glad this story had a happy ending.
These kinds of scenarios are why I built Relica [1]. It backs up to local disks, network drives, remote computers (on LAN or anywhere with a public IP address), your own cloud storage, or our own special formula we call the Relica Cloud: one upload, five independent cloud providers -- replicated in real-time.<p>And restores can use the open-source tool restic [2], so you don't have to be locked into Relica for accessing your data.<p>We're working on the ability to do byte-for-byte copies of a repository to other destinations to make data even harder to lose in these kinds of disaster scenarios, as well as a new UI to make it more pleasant and powerful.<p>Anyway, our goal is help make robust backup strategies like what this guy needs really, really painless, because I'm as paranoid as he should have been about losing data.<p>[1]: <a href="https://relicabackup.com" rel="nofollow">https://relicabackup.com</a><p>[2]: <a href="https://github.com/restic/restic" rel="nofollow">https://github.com/restic/restic</a>
>I don’t know what happened for sure, but I think it may have been a power surge that fried the boards on both the Synology and the USB, as they were plugged in to the same socket.<p>He didn't have a surge protector? Sweet Jesus. I don't plug my <i>backed-up</i> PC into anything not surge protected.
It's a common scenario, unfortunately. Synology RAID is nice. External USB on the same power circuit is not a good idea. Actually, I'd look at least into a surge protector. But it's quite possible to have those blow up and still fry your hardware.<p>I'd run some batteries in the basement and run my Synology (or similar) off of those. Additionally, the USB Backup of backups should be in the garage or the attic, if at all possible. Also, a nice cloud-backup solution that is capable of delta-uploads is a very good idea (cover against fire in the house, or any other "catastrophic" failure).<p>If you don't have DC experience or if you didn't do much hardware, it's common to over-focus on software. And to be fair, vice-versa :)
I had a similar experience with a relative's Seagate NAS.<p>Except the ext filesystem was unreadable because it used a different page size. Required some shenanigans in userland by thankfully I was able to recover the data. Seemed like a software fault on the box.<p>The chassis had to be destroyed to remove the drive and it was interesting to see the warranty explicitly mention the customer was allowed to do this to recover their data.
You need one copy somewhere else! What if there's a break-in or the roof leaks or whatever? My low-tech solution is two external drives, one of which is at my Mom's place and gets swapped every month or two.
Things that jumped out at me from this:<p>First, the USB external is probably OK except that its USB circuitry has taken a power hit. If it's a standard SATA drive inside that could probably be shucked and accessed. Counter: Some of these have drives that are no longer SATA but have a bunch of the USB connectivity built into the drive. At that point, you'd probably be looking at a few hundred $ of data recovery costs (yes, that little). Professional recovery of the RAID would be more expensive because pricing is often based on the number and capacity of the drives.<p>Second, RAID5? I know these were only 1TB drives, but be very wary of anything with only a single parity disk if you're looking at drives of 1TB or larger, particularly if they're sequentially-numbered drives from the same lot. With modern TB+ drives there's a not-insignificant chance of drive errors as you hammer the remaining drives to rebuild the array. If building one of these now, the price difference between a RAID5 of smaller disks and a RAID6 of larger ones is probably only a few dollars.<p>Third, if actually doing recovery the first thing you want to do is image the drives and work from the images. ddrescue is probably your simplest option there, but yes you're going to need a big chunk of drive space available.
Here is what I do to hedge the risk of electrical failure (raid+usb on the same circuit), fire (both copies in same structure), malicious compromise of my machine and theft:<p>- My workstation has linux software raid-1 of 2x 6TiB drives (this provides robustness and uptime in case of single drive failures and ease of recovery).<p>- Another machine in my garage doing incremental daily backup pulls over the network. It is setup as multiple discrete hard drives thus partitioning single drive failures (low cost, the garage is a separate building, the host machine is an arm board that actually turns off HD PSU when not backing up, so hard drives are fairly isolated from power spikes).<p>- I make a monthly incremental backup (three sets) onto an external 6TB usb hard drive encrypted with luks. This drive spends 99% of it's life powered down in a cabinet at my office at work. It is protected against theft, fire, electrical spikes, etc... by my employer.<p>- The is not a *ucking cloud anywhere in this picture. I can get access to my backups within ~1hr in worst case (round trip drive to office to pickup my drive).<p>You Kids need to learn how to take care of your shit - now get off my lawn!!
Over Christmas, I backed up all my stuff to a 3TB external hard drive that I took with me when I visited my parents and intentionally left the drive on a shelf at their house.<p>This is what I call a poor man's offsite backup.
Something like that happened to me. I had several TB of data on a RAID10 array on an LSI MegaRAID controller. A nearby lightning strike took out the server. And the server manufacturer had gone out of business.<p>I had backups of the data itself. But I'd been doing lots of data massaging, and didn't have enough storage to keep copies of every step.<p>Anyway, so I bought a couple new servers. One to replace the dead one, to be setup with SQL Server. And a low-end one that would accept the controller from the old server. I just left the drives in their cage, and jury rigged power and data connections.<p>And it worked.
Couple of observations.<p>First of all, given the price of storage, for backups, I don’t think anything other than mirroring makes sense. Just get 2 big hard drives for your NAS and set them up for mirroring. In the event of a filmier, you can read directly.<p>Next, you don’t have a full backup without offsite storage. Even if it wasn’t a power surge, there could be flooding or fire at a single location.<p>Always remember the basic 3 2 1 rule!
I've always thought of a NAS as high speed local storage with a little redundancy in raid, but I'd never treat it as a backup solution. Only because then all my data is under one roof. Fire, earthquake, flood or other local event that's big enough and all my data is gone. I'm lazy so I haven't setup anything fancy but backblaze and dropbox do all my off site backup for me. It's very cheap. 10 years of backblaze would be cheaper than a NAS.
Power problems can be both stealthy and deadly.<p>Years ago I had a desktop with a 4-disk RAID 5 where the SSDs failed in quick succession, it's common. I lost some data-- or rather recovered files manually I think from a failed RAID and cold backup, switched to spinning disks, but after a while the new disks started beeping and generating RAID errors.<p>After much time and anxious guessing, I swapped the power supply and never had a problem since.
I recently was also thinking about how to organize my data & backups. I still have not really decided. Esp about the software. I collected lots of options here: <a href="https://github.com/albertz/wiki/blob/master/backup-software.md" rel="nofollow">https://github.com/albertz/wiki/blob/master/backup-software....</a><p>At the moment, I really like Perkeep (<a href="https://perkeep.org/" rel="nofollow">https://perkeep.org/</a>). But I'm not sure whether this is a solution for everything.<p>On the hardware side, I also have not really decided. I want to build up my own NAS (custom hardware, no preconfigured thing), which should be quiet (if it is not doing anything, i.e. most of the time), as it will be in my home. Another NAS maybe at my parents home. And then maybe some cloud storage.
I have looked and looked since 1998, tried rsync, RAIDs, striped drives, ATA over Ethernet, lvm, ZFS, all sorts of things.<p>What finally clicked for me, is <a href="https://www.greyhole.net" rel="nofollow">https://www.greyhole.net</a><p>It's like magic. Decide how much redundancy you want. Then just add drives to it. It balances files automatically across drives. You can have remote drives in the mix.<p>What it is not good at, is many small files. But for my use, media files and backups (tar archives) it's a breeze. And the files are stored as normal file on the drives it distributes too, so there is nothing complicated to dig into should disaster strike. (Not that it has happened to me.)<p>No affiliation, just finally in a Zen state of mind when it comes to my home NAS.<p>Next step - make sure all of that is backed up off site too, but that is another thing altogether...
I had all of my backups on a 2TB external drive, which worked great until the MFT got corrupted somehow. Suddenly all my eggs in one backup basket felt a bit silly. Fortunately I was able to recover all of it. I'm in the process of partitioning out a new backup system to avoid that in the future.
After thinking about this kind of data apocalypse for a bit I realised that the portion of my data that is mission critical is actually really small.<p>Throwing it on multiple clouds with version history isn't an issue.<p>(I'd recommend an O365 sub + duplicati...in theory you can push like 5TB to MS cloud).
Silly mistake I nearly made:<p>MacBook Pro in for repair incl. wiped disk. No problem, I have two external drives with regular TimeMachine backups, so go ahead.<p>Having received the laptop back, I plug in one of the disk drives and - oops, it's encrypted (with a good, safe, long password of random characters), which had previously been stored in the local login keychain, so that the disk drives have always just silently automatically mounted over the years, and I totally forgot that they were encrypted!<p>Fortunately, I had the password saved somewhere and access to it. Otherwise my backups would have been for nought (though of course they themselves had the disk password inside... well protected by itself.)
Is it weird that I don't have GBs of my own personal files/data? I get if you have movies/tv shows or whatever, that's different. I just don't know what personal data you could possibly have that would amount to GBs...
I currently have way to much data lying around for any metered cloud storage to be economical, so I'm currently subscribing to both Backblaze and Crashplan for their unlimited storage backup plans.<p>Crashplan killed off their consumer plan a while ago, so I ended up moving to their business plan at double the price. In my case even at that price it was still worth it considering the amount of storage I'm using.<p>Anyone aware of other services offering unlimited storage for a single user? I know Dropbox, Google Drive, and OneDrive all offer unlimited storage in their business plans, but they all require a certain number of users before the unlimited storage kicks in.
A surge from a lightning strike near my home travelled over the cable line to kill a network switch and the WAN port on a firewall. (Strangely, the cable modem was spared.)<p>Everything was on a decent UPS... but I’d completely forgotten the cable line.
- SSD + HDD in laptop, this latter for storage, because it actually tells me, if it's unhappy and about to die, unlike the m.2 ssd. The cost of this is to have an older laptop, in my case, a thinkpad x250.<p>- synced to home server, which, at this point, is a thinkpad x201 on an ultrabase with 2 disks - it has built-in ups, called a battery<p>- all of this synced to off-site rented server in Germany<p>- irreplacable photos are on blu-ray on yearly archives<p>This covers lost laptop, burglary, house fire/flood, etc. To avoid problems with lost ssh keys, I have a few users on that rented server which can log in with a password, in case of emergency.
My current approach.<p>Personal media archive, Windows 10 Pro, Storage Space with Parity across 4 HDD in a sata jbod. This is a purely software RAID. Moving the drives (or part of them) to another Windows 10 system allows for seamless recovery.<p>This archive, as well as all personal computers use Backblaze for offsite backup (including versioning). Versioning is important in case malware/accidents/buggy software. I don't consider any backup plan complete without this and being off-site (fire/theft)<p>For my business servers I use Tarsnap. (Off-site and versioning, 45 days)<p>Edit: Oh and everything is on UPS
Data Storage is dirt cheap. Buy a 6tb for $100 every few months and throw important folders onto it. Place old ones at relatives homes for offsite.<p>Never worry about this again.
My first thought is "why don't you keep a backup?" I really don't know what the moral of this story is beyond "don't keep one copy of all your data, especially in the same physical location."<p>Online storage is cheap. Bandwidth is cheap. There's a multitude of solutions in the comments if a consumer solution like Dropbox or Google Drive isn't good enough for you.
There isn't really a good excuse not to have an offsite backup, especially when services like back blaze are $5 a month.<p>Also ... RAID is not a backup.
Synology allows syncing of your NAS to a similar Synoogy NAS on another location (Cloud station server). That's what I am doing. This doesn't prevent issues with data corruption, but prevents these kind of issues where you 'complete' NAS fails due to a possible surge.
The failure modes I consider most probable at my home are a whole-house power surge, and a loss of structure (fire, natural disaster). With a 1 Gbps (synchronous) FTTH connection, and a Backblaze subscription, that's about all the peace of mind I need!
I'm surprised it's not mentioned here but a lot of the Synology nas's have had power supply issues in addition to the prior Intel Atom issues. I swapped out four units twice each (1815+'s) and swore off their hardware permanently.
Pretty happy with my solution. Keeps me safe from floods, ransomware, and losing google drive access:<p>- originals: 2TB google drive + 50GB icloud<p>- backup #1 (autosync from drive/icloud): 4TB internal drive<p>- backup #2
(cold storage): 4TB external drive
That’s part of the reason I keep my personal data backed up two two sets of NAS. In addition, it’s mapped as a Dropbox drive on one of them.<p>From my families perspective it’s just one drive, but it’s replicated everywhere.
Me too, because I was storing it on IPFS. IPFS had a minor bug and I've lost all the references to the data.<p>You know, the data is still there, I just don't know what hashes correspond to each objects.
I use Synology cloud software to backup to both Google Photos and Backblaze. Way faster than Glacier, and costs me <$5 month (haven't reached 1TB yet).
After watching Steve from Gamers Nexus talk about their second Synology NAS failure in a single quarter, I instantly figured that OP was using a Synology product. Their stuff must be bad if I'm making that association.<p><a href="https://www.youtube.com/watch?v=K7ly8zde3dE" rel="nofollow">https://www.youtube.com/watch?v=K7ly8zde3dE</a>
Coincidental timing for this article! I just wrote in my own blog yesterday about the level of importance I had placed upon the data I had stored on my PC [0]. (TLDR - Data I thought was unimportant ended up actually being the opposite right after I lost my external drive).<p>[0] - <a href="http://devan.blaze.com.au/blog/2019/1/20/the-folly-of-unimportant-data" rel="nofollow">http://devan.blaze.com.au/blog/2019/1/20/the-folly-of-unimpo...</a>
TL;DR: a power surge killed my PSU and maybe my motherboard, causing me to freak out and btw I don't have any cold storage.<p>I'll bet this guy did not have a surge protector in front of his Synology PSU.
I've been working with computers for so long that total failure is not a probability, it's a certainty. Last year, in fact, my desktop burst into flames. I've had smoke come out of desktops several times before, but that was the first flamer.
The best thing i found to enforce backup discipline on myself is to regularly migrate between machines- who are regularly offline for more then 4 hours.<p>You will have mostly current backups of your projectdata and workdata on all those machines you migrate from and towards.<p>The problem theire then becomes destructive automation. You must avoid automating syncs with half-corrupted or full-corrupted instances of your work environment.<p>Also backups. Always backups.
He doesnt confirm that the drives are faulty, and it isnt clear that they are, as the data is accessible while maybe some partition or boot data was scrambled but repairable<p>The NAS should also have a warranty of some kind or the controller could be repaired for cheap<p>He was never in any data peril, so just fix that and then add an offsite backup to the mix
M Discs brother, they are cleared to last 1000+ years. Long after most people's cloud backups have turned to dust, my most treasured data will endure due to my use of these discs (which may well be unreadable from a hardware standpoint in 3019. But shit, hope they keep the schematics around.
Inspired by this post, I decided to document my backup strategy and shortcomings here:
<a href="https://natalian.org/2019/01/20/Data_I_would_not_like_to_lose/" rel="nofollow">https://natalian.org/2019/01/20/Data_I_would_not_like_to_los...</a><p>tl;dr I'm trusting Apple.
I just don't understand rolling your own backup drives in 2018 unless you're up to something nefarious. I rest peacefully knowing my data will never be lost by Dropbox, and the immediate backup means I don't have to mess with slow, periodic all-at-once backups.
If you have 4TB of data saved at home you are a hoarder. Why do you have that much stuff. I don't have any data at home that I would miss very much if I lost it.
I see some terrible backup strategies here.<p>1. Backups should not be on a single drive.
2. Backups without checksums will result in corruption.
3. Offsite is a must.
4. Unencrypted off site backup means someone already copied your data.
5. Encrypted offsite backups should have forward secrecy. So different keys for each file and keys file gets backed up encrypted.<p>My backup strategy:
File server runs zfs raidz with Daily/weekly/monthly snapshots on disk.<p>Snapshots get copied to 2 external drives, zfs mirrored.<p>Files get encrypted and uploaded to backblaze using my custom software. Nothing fancy, just standard authenticated encryption (chacha20poly-poly1306) but with per file key management and argon2.