I'm curious to hear of experiences running distributed filesystems like glusterfs or ceph (others too?) on Digitalocean's block storage.<p>I've got a side project that will be continuously adding many 50kb to 20mb files, so easy expansion is necessary.<p>Any "at-scale" (ie. production, under real load) performance reports/benchmarks appreciated.<p>Reliability, recovery from node failure, etc comments welcome.
Don't use either. I tried and failed with both. Expensive hardware and a PhD required to go through all the config options. When I did get both to work, they were very slow and I kept getting replication issues and bricks that wouldn't heal.<p>This cost two months of my life and I was not flavur of the month with my employer.<p>I eventually tried lsyncd ( <a href="https://axkibe.github.io/lsyncd/" rel="nofollow">https://axkibe.github.io/lsyncd/</a>) and it works like a charm. It took me about a week to get a decent setup going, but it has never let me down since, can handle large numbers of small files (similar to yours) and smaller numbers of large files. I use unison ( <a href="https://www.howtoforge.com/tutorial/unison-file-sync-between-two-servers-on-debian-jessie/" rel="nofollow">https://www.howtoforge.com/tutorial/unison-file-sync-between...</a>) for web server pairs.
Sorry for Unison, I should point out that I use fcron for running Unison every 5 seconds. This on average syncs 15-20 20Mb files every 5 seconds without any issue. This is similar to what you describe in terms of load.