Half a petabyte in storage and 3 TB transfer a day? Shit. That's nothing, unless you're "saving money by using the cloud".<p>While I feel for them, they put themselves in this position. Even when someone gives you something apparently for free, you have to consider what might happen when it disappears, and it seems that nobody thought about it, or if they did, they didn't care.<p>People often learn how expensive "the cloud" is the hard way.
Cloudflyer (<a href="https://www.cloudflyer.io/" rel="nofollow">https://www.cloudflyer.io/</a>) offers free migrations from S3 to Storj. Storj is $4/TB/mo, $7/TB egress. It's decentralized storage, so the base functionality has CDN-like performance (no Cloudfront needed), and you don't need to pay extra for multiregion redundancy.<p>I jumped on the Nix Foundation Matrix to try and help them directly, but for anyone reading this thread, Storj might be able to cut a zero off your storage costs. Check us out!<p>Full disclosure - I work for Storj. <a href="https://www.storj.io/" rel="nofollow">https://www.storj.io/</a>
One of my colocated servers has 60TB of space to spare and has a 10gbe unmetered uplink on the provider's IP-transit blend.<p>It is easy to configure traffic shaping on the box so that any unused upload bandwidth can be used without any impact to existing applications.<p>I would be happy to seed NixOS torrents. There is an extension to torrents where files can be added to a torrent after a torrent is released that might make sense to look at.
Google and some napkin math tell me that this kind of workload should not be on the cloud.<p>A 4U or 8U rack server with 1 PB of storage and plenty of cores: ~$40K<p>Colo facilities in the Bay Area: $400-500/mo<p>Compared against the status quo of $9K/mo, the ROI ends up being 5 months per box, for twice the storage capacity they need. Set up multiple of those boxes around the world to the extent that you care about redundancy/distribution, and they still end up far ahead in a few years time.
S3's costs are insane for this use case, can't imagine using it to for a Linux distribution's repositories.<p>edit: Oh man, there's potentially a $32k migration fee to pay for S3 egress, as well.
The only thing I wonder is how much space it would use if the nar files were stored without .xz and in a store that does smart deduplication. Because of the way Nix works, there's going to be a lot of e.g. ELF files that are nearly identical except for the rpath.
2 days ago someone said they had thousands of dollars of AWS credits to use.<p><a href="https://news.ycombinator.com/item?id=36149966" rel="nofollow">https://news.ycombinator.com/item?id=36149966</a>
I've never heard of LogicBlox but they were sponsoring $9000 a month? That's crazy, normally even big sponsors will give a once-off $5000. Well, well done to them
That seems like the perfect use case for IPFS. I can even bet that if the people at the NixOS foundation played their cards right, they could get the Filecoin bag holders to <i>pay</i> for the development of this new feature.
Can someone please enlighten my uninformed self why NixOS needs to store this mindboggling amount of data and why it would be damaging to the research community specifically if it were to be lost? 9000$ a month is a LOT.
“That would incur a fixed cost of around $32k for the migration.”<p>They should consider Backblaze b2 if they want s3 api compatibility. Backblaze currently offers to cover the migration costs assuming some spend commitment.
Personally, I'll shovel money to the TVL folks for tvix-store before I give a single penny that might end up in Amazon's pocket for S3 hosting...
This seems to be about a cache of compiled stuff where it'll cost $32k to move the data out of S3. So... rebuild the cache somewhere else and drop the original data?
There has been a controversy around the server cache.nixos.org. The signing ed25519 cache key is from the year 2015 and there is no plan to rotate the key. It is apparent that a key cannot hold for all these years.<p>I have been made aware of the matter when somebody on the matrix claimed that there is no plan to rotate the cache key. He/She said reasons behind that are explained by Eelco D. in <a href="https://www.youtube.com/watch?v=yXPQdZh8o7c">https://www.youtube.com/watch?v=yXPQdZh8o7c</a>. Unfortunately, I couldn't confirm this. That person afterwards claimed that the video is cut and that part was removed intentionally.<p>If that's the case, it might be the reason why ppl stop paying for the cache server costs. Furthermore, if the foundation attempted to hide the facts by cutting the video and re-uploading, for me, this is a matter of dishonesty and somebody should take the responsibility for it. I believe this person is Ron E., the current director of the Nixos Foundation. But in any case, this claim should be backed by finding the original stream of the Nix 20th Community Panel recording.<p>For the rest of us, until some cryptogeek fixes nix to support certs and key lifecycles, the solution is to build a local cache with hydra and flakes. There are many ppl keen on the subject, you can look for one on the matrix.
I can recommend Hetzner Storage boxes <a href="https://www.hetzner.com/storage/storage-box?country=us" rel="nofollow">https://www.hetzner.com/storage/storage-box?country=us</a> its quite cheap (€40.60 for 20 TB).<p>Just order 25x BX41 Boxes with 500 TB for 1015€. They do not support S3 protocol, but have plenty of other protocol options (FTP, SFTP, HTTPS, rsync, mount as network drive, ...).
Any cloud provider or CDN that agreed to just sponsor such an important project would gain so much community love that I think they should just do it. Cloudflare should just do an agreement special with Nix and store everything for free. The cost would be not negligible but the reputation benefit and goodwill will be immense.
> Migrating the buckets to Cloudflare R2 52, and using the generous OSS sponsorship offer they recently announced 89. That would incur a fixed cost of around $32k for the migration.<p>I assume this is do to egress from S3. I wonder how that cost compares to re-generating the historic caches. Presumably that is possible with nix.
We will be holding the community call on Wednesday June 7th at 5pm (GMT+2)
Planned Agenda
* Brief budget and timeline review
* Review/discuss all potential options
* Brainstorm/Q&A<p>Info to join is in the original discourse post.
Thank you again for jumping all in on this with us!
Maybe explore Wasabi, costs $5 per TB/month with no egress costs
<a href="https://wasabi.com/cloud-storage-pricing/#three-info" rel="nofollow">https://wasabi.com/cloud-storage-pricing/#three-info</a><p>Can get cheaper as well if you go for reservered capacity model.
I’m surprised aws doesn’t get in front of this and offer the s3 storage as a donation. It would be tax deductible at retail price, and generate good will. In fact I always thought they should open up storage for popular distributions of Linux and stuff like brew etc.
I don't get why these things aren't designed around bittorrent, at least as first line of defence.<p>Also means the most popular packages have the most seeders<p>I bet it is easier to get people to seed a bit than to get them to pull out a credit card
When you reach scale and need a CDN, DIY on bare metal on colo and roundrobin using AnyCast.<p>And then find out why you're consuming so much bandwidth and try to use less.
> That would incur a fixed cost of around $32k for the migration<p>The EU needs to get onto banning/reducing lock in like this by making migration expensive. All the cloud providers clearly have pricing models to try to lock customers from moving elsewhere.
I would suggest them a service like wasabi.com. There they wouldn‘t pay the egress traffic only the used storage. But yeah probably the best solutions would be cloudflare if they manage to get the OSS sponsorship.
S3 is a highly reliable storage media. You should use a CDN that transparently caches S3 artifacts to save on reads, such as AWS CloudFront or Cloudflare R2 are good choices.
tl:dr operating yourself around 6-10k€/year fixed price for hosting the 500TB.
If cloudflare offers to store that and operate it for free this is a no brainer for cloudfare R2, if the 32k migration is really a one time only cost.<p>i personally would invest in a solution which decreases or at least flattens the cost, when the storage needs increase, instead of the current linear curve. Biggest problem in "cloud" and biggest win in hosting yourself.<p>if you want to do it yourself i would probably think about something like this:<p>500TB with 20tb hdd's are around 28 disks needed.
This will cost between 15-20.000€ per server(4U with sata/sas controller and 36-40 disk chassis)<p>with no raid and high availability you will need 2 locations and 2 servers in a colocated setup. i would say colo will cost you 500-1000€ a month<p>Then you would use the cloudflare cdn with their OSS offering and be done with it.<p>initial cost 40k
operational cost + colo = around 6-12k/year i would assume, if you have people nearby to do maintenance.
What about looking at Wasabi? It’s $5.99 USD + (any applicable taxes) per TB per month for S3 compatible storage on the pay as you go plan
<a href="https://wasabi.com" rel="nofollow">https://wasabi.com</a>