Hi HN,<p>I'm looking to build something to transfer files between two devices on a network (one client and one server, both of which are under my control).<p>Obviously I could write something bespoke with raw TCP sockets, but rather than reinventing the wheel I'm curious about what existing options people recommend. I assume there are some better technologies than FTP nowadays?<p>Ideally some kind of built-in fault tolerance would be great, because my plan is to use this on a phone/in an environment where the connection could be interrupted.<p>Edit: just to clarify - this is something I want to build into an application I am writing, ideally with support across iOS (client), Windows, and mac (server).<p>One way transfer is all I need, and I mostly plan on photos/videos (so multiple files ~3-20MB in size).<p>Thanks!
I use syncthing for this. It's a little fiddly to set up, and transfer speeds aren't great. However, it's very reliable once configured, and it barely uses any resources after an initial scan of the folder you want to sync.
<a href="https://iroh.computer/sendme" rel="nofollow noreferrer">https://iroh.computer/sendme</a><p><a href="https://iroh.computer/" rel="nofollow noreferrer">https://iroh.computer/</a><p>Iroh is what you’re looking for. You can embed into your code, and it handled wire encryption, verification, and NAT hole punching. All over QUIC.
I highly recommend Unison (<a href="https://github.com/bcpierce00/unison">https://github.com/bcpierce00/unison</a>)<p>It allows you to sync between 2 machines (bi-directional) over TCP or SSH.<p>Note that TCP way is not encrypted, you may use wireguard as transport layer encryption for that purpose...<p>You can use an external application to copy if file size is larger than an arbitrary number. (Eg: use rsync for files > 1gb)
seconding rsync and syncthing.<p>the server could expose an smb or nfs share, the client could mount it, and then sync to that mount.<p>rsync over ssh also works, if you do not want to run smb/nfs.<p>this is also a cool tool <a href="https://rclone.org/" rel="nofollow noreferrer">https://rclone.org/</a>
On a similar note, can someone tell me what's the fastest (wireless) way to transfer files between two laptops on same network (i.e. hotspot)?<p>scp, rsync, wormhole give me only 2-3 mb/s.<p>For the context, I'm trying to transfer about 50-70 GB files.<p>What's causing the bottleneck here and what am I missing? Thanks in advance!<p><a href="https://github.com/magic-wormhole/magic-wormhole">https://github.com/magic-wormhole/magic-wormhole</a>
If it’s one way (that wasn’t quite clear from the requirements to me).<p>take a look at <a href="https://tus.io/" rel="nofollow noreferrer">https://tus.io/</a>
I built something on top of the Syncthing API this week after using it on its own for years.<p>A local instance of Syncthing can behave as a robust sync tool + inotify API for applications consuming the files: <a href="https://docs.syncthing.net/rest/events-get.html#get-rest-events-disk" rel="nofollow noreferrer">https://docs.syncthing.net/rest/events-get.html#get-rest-eve...</a><p>i believe there's an embeddable golang library, but if you want something easy to use on android check in on syncthing-fork which lets you define more granular sync conditions including "just turn on 5 minutes every hour" <a href="https://github.com/Catfriend1/syncthing-android">https://github.com/Catfriend1/syncthing-android</a>
What are your latency and bandwidth requirements? How big are the files? If you are already looking past obvious TCP-based choices like HTTP and FTP, you might be interested in FASP/Aspera <a href="https://en.m.wikipedia.org/wiki/Fast_and_Secure_Protocol" rel="nofollow noreferrer">https://en.m.wikipedia.org/wiki/Fast_and_Secure_Protocol</a><p>Edit: I’ll leave it here just in case it is useful for others but it may or may not be embeddable into your app, especially on the phone.
Grab an S3 bucket on amazon.<p>Do a 3 way sync with the s3 command line tool.<p>That way, you have a neat cloud backup as well. Wouldn't take any more than 20 minutes total to set up.
Use HTTP. For fault tolerance, use resumeable downloads or resumeable uploads. There is work at the IETF on resumeable uploads right now: <a href="https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumable-upload/" rel="nofollow noreferrer">https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumabl...</a>