Incremental Backups Using GNU Tar and S3

133 点作者 cirowrc超过 7 年前

14 条评论

rsync超过 7 年前

If you're intrigued by using basic unix primitives for tasks like this you'd probably also be intrigued by a cloud storage product that was built to act like a unix primitive.[1][2]If you're interested in point-in-time snapshots, you're probably also intrigued by our ZFS platform that gives you day/week/month/quarter/year snapshots that you don't have to configure or maintain - you just do a "dumb" rsync (or whatever) to us and the snapshots just appear.If you're interested in encrypted backups you should look into the 'borg backup' tool which has become the de facto standard for remote, encrypted, changes-only-upload backups.[3][4]Finally, if S3 pricing is important, you should email us about our "HN Readers'" discount.[1] <a href="https://www.rsync.net" rel="nofollow">https://www.rsync.net</a>[2] Examples: <a href="http://www.rsync.net/resources/howto/remote_commands.html" rel="nofollow">http://www.rsync.net/resources/howto/remote_commands.html</a>[3] <a href="https://www.stavros.io/posts/holy-grail-backups/" rel="nofollow">https://www.stavros.io/posts/holy-grail-backups/</a>[4] <a href="http://rsync.net/products/attic.html" rel="nofollow">http://rsync.net/products/attic.html</a>

评论 #15832946 未加载

评论 #15833011 未加载

评论 #15834811 未加载

loeg超过 7 年前

You can do much the same thing with less work using the Tarsnap tool and service. Rather than incremental backup, blocks are deduplicated on the client and only unique blocks are backed up. This has about the same storage efficiency as incremental backups, but has the benefit of not relying upon "full" backup plus incremental diffs to achieve the final snapshot contents.(I have no affiliation with Tarsnap other than Colin seems like a nice guy and I am a customer.)I have full backups configured every 12 hours. Just for example, on my last backup, the total logical size was 12 GB; the compressed size of the undeduplicated blocks was 7.2 GB; the total size of actual new data, uncompressed, was 181 MB; and the final sum uploaded for this new full backup was 72 MB. Logically, Tarsnap stores 2.9 TB of my backups, but after compression and deduplication the "physical" requirement is only 16 GB.For this I pay about 17¢ USD/day, or $62/year. I could probably try to lower my use storage use somewhat (the largest component of that cost, 13.4¢/day) but it hasn't been worth my time yet.

评论 #15833577 未加载

funkaster超过 7 年前

I used to have a "poor man's time machine" system based on rsync + hard links to files that didn't change with new backups. Essentially it was the same concept than time machine. Of course you couldn't upload a single "snapshot" because tar wouldn't know what's a hard link. One advantage of using rsync is that you can also keep track of things you delete.Today I'm using zfs with real snapshots. For systems with no zfs support (my wife's iMac for instance), I have a zfs fs that those systems rsync to, after the rsync is done I create a snapshot. All scripted. The snapshots can be stored in another server for an additional layer of backup, or incrementally send them to s3 if you want.

评论 #15832814 未加载

评论 #15832873 未加载

评论 #15832720 未加载

评论 #15832845 未加载

评论 #15832694 未加载

magnetic超过 7 年前

If you have a large amount of data to backup, cloud storage may be too expensive.My data set is about 8 TB (my wife is a professional photographer), and it would be too expensive to keep in S3, so I have an "offsite backup system" that is hosted at my in-laws. It's just a RaspberryPi + an 8 TB drive encrypted with LUKS (in case it gets stolen or tampered with).Every night, the RPi syncs the data from my house with rsnapshot (which is a TimeMachine like tool that uses hard links with rsync over ssh).Because of how rsnapshot works, I can always go there and look for a file in one of the directories: it looks just like the original hierarchy, and I can just browse the filesystem as I would on the original system.I also don't have to "trust a 3rd party" that the data is really there. I remember some posts on HN about people who used some backup services successfully... until restore time. I'm always cautious about the magical "set it and forget it" service that is a black box.The first sync has to be done "on the premises" because of the sheer amount of data to transfer, but then the few daily Gigs of data can easily go over the net while everyone is sleeping.

评论 #15834501 未加载

评论 #15835956 未加载

rasengan超过 7 年前

I would really recommend using duplicity[1]. It supports gpg, incremental backups and more.[1] <a href="http://duplicity.nongnu.org/" rel="nofollow">http://duplicity.nongnu.org/</a>

评论 #15838309 未加载

评论 #15833248 未加载

评论 #15834172 未加载

评论 #15833489 未加载

arca_vorago超过 7 年前

Tar works for small things, but over an arbitrary file number file size combo it becomes unwieldy. Rsync, rbackup, rsnapshot, are great tools based on rsync, and these days Borg has been getting a lot of traction. Bacula is great but complicated to setup and manage. There is a newer very interesting one called Ur that seems promising... And my mind is blanking on some of the others, I'll comment when I'm not mobile so I can look at my list.Also don't forget zfs/btrfs functions that might be relevant.

评论 #15832643 未加载

bauerd超过 7 年前

Worth checking out for backups is git-annex with remote bup repositories:* <a href="https://github.com/bup/bup" rel="nofollow">https://github.com/bup/bup</a>* <a href="http://git-annex.branchable.com/special_remotes/bup/" rel="nofollow">http://git-annex.branchable.com/special_remotes/bup/</a>

评论 #15832385 未加载

terrik超过 7 年前

A great, though non-free, backup tool that supports incremental encrypted backups is Arq[0].Arq supports painless back ups to multiple cloud storage providers, with budget enforcements, etc.[0] <a href="https://www.arqbackup.com" rel="nofollow">https://www.arqbackup.com</a>

muxator超过 7 年前

With Btrfs or ZFS the whole concept of incremental backup is reshuffled: basically you have consistent full backups, only occupying the space of a full backup + all the diffs, and you only transfer the changed blocks. It's really something with no comparison to other methods.While I love ZFS, let's not forget that btrfs has its strengths (a lot more flexibility, mainline support), and, provided that your use case is single disk, Raid 1 or raid 10, has been working super reliably for some years now.

aedocw超过 7 年前

I recently wanted to get backups going in a similar way (incremental with copies on-site and off-site). I ended up using Borg and rclone, sending the backups to s3 compatible Wasabi storage. It's been working great, and coupled with zerotier on machines that are not in the house (family members), I ended up with a pretty resilient system that only costs a few dollars per month. I wrote up the details here: <a href="https://localconspiracy.com/2017/10/backup-everything.html" rel="nofollow">https://localconspiracy.com/2017/10/backup-everything.html</a>

boramalper超过 7 年前

I'm not sure if tar is a good format for backups, due to its extremely sequential nature (e.g. you can't get a list of the files in an archive without scanning the whole archive...)

评论 #15832292 未加载

评论 #15832702 未加载

IgorPartola超过 7 年前

Note that one point of failure of this setup is that you are keeping the snapshot/index filed locally. Ideally, you want to back those up as well.

atomi超过 7 年前

I use restic to minio stored on a raid1 btrfs system and I love it.

ape4超过 7 年前

You'd definitely want to do a full backup occasionally.

评论 #15832389 未加载