I've often heard that the best way to send large amounts of data is to actually send it via snail mail. But if that is not an option what is the best way to send, say 500GB, over the internet? Past Experience preferred but creative solutions welcome!
If, for some reason, you are averse to rsync (there are plenty of Windows clients), then bittorrent is second-best: it works just as well for transferring private data, just turn off all the metadata broadcast options and make sure that encryption is required. (Not so easy for incremental transfers, though)
I would use scp with the -C option to compress the data for the first attempt to transfer the file. Any subsequent attempts to transfer the file should use rsync (this includes any updates to the file).<p>Another option is to use a cloud data storage provider (Dropbox, Box, Google Drive, etc) and install their software that keeps files in sync. Then you can just put the file in a local folder and let their software sync it to the cloud. After sync is complete, you email a link to share the file. Of course, their software is probably just a GUI for rsync.
I assume you're transferring between two hosts you both control. Some sane options are:<p>- rsync<p>- SFTP-over-SSH, or SFTP over a dedicated VPN<p>- private, encrypted torrents
Aspera is often used in the film industry to move large video files around. My understanding is their software takes over some layers of the network stack to make sure the pipe stays saturated.<p><a href="http://asperasoft.com/" rel="nofollow">http://asperasoft.com/</a>
Do you need to transmit 500GB every time or just a diff from a previous dataset ? If it's the later case, using the send/receive functionality of a file system with snapshot and incremental backup (ZFS,BTRFS etc..) can be significantly faster than using pure rsync. Rsync would needs to scan the complete 500GB of data to find the blocks to send, while send/receive can compute the diff much faster
More details please.<p>* Are we talking a few giant files or thousand of small-ish files?<p>* Who is on the receiving end? Technical people from who we can expect that they are able to run some command-line stuff or your average joe?<p>* What type of scenarios must the solution work? What OSes? Is it acceptable to install extra software or must it work out of the box?<p>etc. etc. etc.
btw not relevant in your case, but a friend working for a large video company regularly drives a truckload of 10 TB tapes around the continent - bandwidth is still an issue at <i>that</i> scale.