Rclone: rsync for cloud storage

Looks really good. I am pleased more projects are adding Google Cloud Drive support now. What I really want to do is:<p>- create documents on my Mac which autosync to Cloud Drive in encrypted format (this should tick that box)<p>- be able to access said documents on any device including iOS, which transparently handles the encryption<p>The use case is I now scan all my documents into PDF format, but keeping them secure and accessing them on iOS seem to be almost mutually exclusive.<p>I looked at some other solutions for this which had their own iOS app and security mechanism (Boxcryptor mainly) and I didn&#x27;t like it - I just didn&#x27;t feel in control. And I got nervous about what happens if Boxcryptor goes under; I don&#x27;t want to rely on them keeping their app up-to-date to read my documents.<p>I know Apple will never allow it but wouldn&#x27;t it be nice to be able to mount your own network drive which all apps could access.

Unfortunately, it appears that binary diffs are not supported.<p>This is a really important aspect for many workflows dealing with large files (like TrueCrypt containers). Contrary to what is stated by the rclone developer [1], at least Dropbox supports binary diffs [2].<p>This should be looked into, at least for Dropbox.<p>[1] <a href="http:&#x2F;&#x2F;rclone.org&#x2F;faq&#x2F;#why-doesn-t-rclone-support-partial-transfers-binary-diffs-like-rsync" rel="nofollow">http:&#x2F;&#x2F;rclone.org&#x2F;faq&#x2F;#why-doesn-t-rclone-support-partial-tr...</a><p>[2] <a href="https:&#x2F;&#x2F;www.dropbox.com&#x2F;en&#x2F;help&#x2F;8" rel="nofollow">https:&#x2F;&#x2F;www.dropbox.com&#x2F;en&#x2F;help&#x2F;8</a>

FWIW: tarsnap is also rsync for cloud storage and Colin (guy who founded and runs tarsnap) also has won a putnam award for his work in mathematics and crypto.<p><a href="http:&#x2F;&#x2F;www.tarsnap.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.tarsnap.com&#x2F;</a>

This fills a real need for me. It does nearly everything I want.<p>Aside from the program itself, your documentation is really good, and special +1 for documenting the crypto thoroughly (and another +1 for using NaCl&#x27;s building blocks in a safe way).<p>As a related point, I recently bought a Chromebook (still unopened), which pushes you heavily towards storing your files in Google Drive. It makes me uneasy to store certain things unencrypted, so I&#x27;ll investigate writing a compatible implementation for ChromeOS.

Rclone is great. I wrote an integration to use it with git-annex (<a href="https:&#x2F;&#x2F;github.com&#x2F;DanielDent&#x2F;git-annex-remote-rclone" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;DanielDent&#x2F;git-annex-remote-rclone</a>).<p>Some of the supported providers (e.g. Amazon Cloud Drive) have a reputation for days-long service outages. Some users of Amazon Cloud Drive have even reported files going missing on occasion.<p>But the great thing with git-annex is you can have your data on multiple clouds (in addition to being on your own equipment), so partial or complete loss of a cloud provider does not need to result in availability or durability issues.

Very cool program.<p>s3s3mirror [0] is another tool for copying data between S3 buckets or the local filesystem. full disclosure: I am the author.<p>At the time I wrote it, I only needed to work with AWS, and needed something very fast to copy huge amounts of data. It works like a champ, but I do think about what it would take to make it cloud-independent; it wouldn&#x27;t be easy to maintain the performance that&#x27;s for sure.<p>[0] <a href="https:&#x2F;&#x2F;github.com&#x2F;cobbzilla&#x2F;s3s3mirror" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;cobbzilla&#x2F;s3s3mirror</a>

How does this compare to duplicity ( <a href="http:&#x2F;&#x2F;duplicity.nongnu.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;duplicity.nongnu.org&#x2F;</a> ) ?

This looks awesome. I&#x27;ve made several attempts at something that could write encrypted files with obfuscated file names to several backends but never ended up with something I was happy with.<p>I&#x27;ll definitely give this a try.<p>Edit: One feature I would like would be to split files into n chunks to obfuscate the length of files (assuming it wasn&#x27;t obvious which chunks go together to make up a file), so instead of a 1:1 relationship there was a 1:n for large files. I suspect this is a lot more work though...

Looks promising, but I&#x27;m not sure about the crypto-part. Can someone give some notes about the security of NaCl Secretbox using Poly1305 as authenticator and XSalsa20 for encryption?<p>Is it justified to assume that this is adequate crypto as long as the nonces are choosen correctly (= as random as possible) and the keysize is bigger than 128bit (rclone uses 256bit key derived from user password)?<p>Documentation of the crypto part can be found here: <a href="http:&#x2F;&#x2F;rclone.org&#x2F;crypt&#x2F;" rel="nofollow">http:&#x2F;&#x2F;rclone.org&#x2F;crypt&#x2F;</a><p>EDIT: added constraint regarding keysize.

Neat. I wrote my own hacky little Python app to upload to dropbox, but they recently broke that with changes to the dropbox python library. I hadn&#x27;t bothered to fix it :)<p>I&#x27;ll check this out instead - thanks for sharing OP.

If you use Amazon Prime, you also get unlimited AWS cloud drive storage for photos. rclone works well backing up all your photos to Prime.

Will this work with any remote host over SSH ? All of the example targets (S3, google cloud, etc.) are things that you <i>can&#x27;t</i> rsync to.<p>That is, can you point it at rsync.net (or your own server that is only running ssh) ?<p>If the author is here, please email us (info@rsync.net) if you&#x27;d like a free account to test with.

Does anyone know if rclone preserves Linux File Permissions regardless of the cloud storage?<p>It&#x27;s not in the feature list and my guess is that this would be hard to implement if you can&#x27;t take assumptions of the underlying file system.

I&#x27;m not sure that syncing to cloud is really the best for most personal users, at least not anymore. If you have multiple devices and use SyncThing &#x2F; etc to sync between them, you&#x27;re protected against device loss and damage without having to put your personal files on a server controlled by someone other than yourself.<p>I was about to install it anyway, but I saw that it doesn&#x27;t have bi-directional sync. If 3 people at work shared a google drive folder and they all tried to sync to it, it sounds like whoever synced last would always win and it could potentially delete &#x2F; alter some file.

I have this running on a Raspberry Pi and it&#x27;s working 24&#x2F;7, uploading media files from my NAS to my Amazon Cloud Drive.<p>I use Kodi on a separate pi to stream the content from Amazon, thereby freeing up space on my NAS.

Worth mentioning Google Cloud Platform supports copying&#x2F;syncing buckets from S3.<p><a href="https:&#x2F;&#x2F;twitter.com&#x2F;gregsramblings&#x2F;status&#x2F;770804483798421504" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;gregsramblings&#x2F;status&#x2F;770804483798421504</a> <a href="https:&#x2F;&#x2F;cloud.google.com&#x2F;storage&#x2F;transfer&#x2F;create-manage-transfer-console" rel="nofollow">https:&#x2F;&#x2F;cloud.google.com&#x2F;storage&#x2F;transfer&#x2F;create-manage-tran...</a>

Has anyone had success with Amazon Drive? 60 USD for unlimited storage or just 12 USD unlimited storage using stenography is hard to beat. If it works better for backup than Backblaze or Crashplans terrible clients and horrid performance it would be a good alternative.

I have been using this successfully with Google Cloud Storage and our own internal Swift object store.<p>For the latter, it uploads much faster than the shell scripts I had been using, and it has similar utility as an &quot;rsync for the cloud&quot;.

I genuinely don&#x27;t understand the use case for this since, for example, Dropbox already just syncs the changes you make to a file and not the whole file, automatically and does so bidirectionally, which this tool does not. So, if anyone can help state more clearly what this is adding over and above the features that the various cloud storage vendors already provide I would benefit from the explanation.

It would be interesting to set this up on FreeNAS. Developers should look into providing this as a plugin to FreeNAS users.

I have used it with a cheap VPS (Ovh. I will test it soon with Scaleway) and it worked fine transferring data between Google Drive and Amazon drive. ;) ps: I did not tested it with encrypted files as it is a few weeks ago option. pps: see also reddit datahoader board for examples. ;)

Does rclone support multi-threaded&#x2F;multi-processing in a similar way to how gsutil support it via the -m option? As a clarification I&#x27;m referring to an equivalent to &quot;gsutil -m rsync ...&quot;<p>I haven&#x27;t been able to find anything in the documentation mentioning this

I know it. :) I use i to off-site my asymmetrically encrypted backup (created with <a href="https:&#x2F;&#x2F;github.com&#x2F;dpc&#x2F;rdedup" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;dpc&#x2F;rdedup</a> ) to Backblaze B2 .

Will this work with two different s3 buckets (2 different regions)?

Been backing my music up to Google Drive with this in a cronjob for past year.<p>Works a treat.... highly reccomended!

I use this to regularly back up to Google Drive, once you link it up, it&#x27;s seamless!

use this in production since one year, installed on a synology nas to backup on ovh storage. please get the github version as the download on the website is quite different

Exactly what I was looking for! Thanks!

No softlayer?

Does anyone else work mainly with Linux but use Google Drive?<p>95% of stuff I work with is Linux but that last 5% is done in Windows for work. I use Google Drive but the lack of syncing is really annoying. I also have a NAS that runs Linux that I would love to use to sync my GDrive&#x2F;Amazon Drive to.<p>I&#x27;ve been brainstorming ideas including but not limiting to seeing if I could use W10 IOT for the RPi and install Drive on there (Pretty sure its impossible).<p>It boggles my mind there isn&#x27;t a elegant solution to this that doesn&#x27;t require me to pay for a service.

Another thumbs-up. I&#x27;ve been using it to mirror to Amazon Cloud Photos (now called Amazon Drive I think) and it&#x27;s rock solid.