Are Package Registries Holding Open-Source Hostage?

FTA: &quot;Ultimately, package registries need to align their incentives with those of maintainers.&quot;<p>Putting it all on the registries to come up with a viable business model and provide this to maintainers without any responsibility[1] on the part of the maintainer seems really one-sided.<p>It costs quite a bit of money to run something like Docker Hub or NPM. If you want something aligned software maintainers first and foremost, you want a non-profit &#x2F; foundation that&#x27;s got priorities aligned with the larger community and not a for-profit entity that has to justify keeping the lights on.<p>Kinda silly headline, too. There are many package registries, but we only see two here that have business models interfering with distribution of software. Only one that&#x27;s really impeding the ability to host software elsewhere if you don&#x27;t like their business model.<p>Docker Hub&#x27;s rate limits seem unlikely to impact most usage of Docker, and people who&#x27;re pulling 200 images every six hours should either seek to set up their own registry to take the load off Docker Hub or throw some money to help shoulder the costs. Even if the user&#x27;s only grabbing Alpine images at 5MB per image, 200 in six hours starts to add up!<p>[1] Granted maintainers may do a lot of work in actually maintaining the software.

This article ignores the fact that for many languages, the package repository is maintained as free software by volunteers (sometimes with funding from a foundation). This includes Perl, Python, Ruby, Rust, and many others.<p>NPM is the odd one out, really. I don&#x27;t think letting one company control a language ecosystem&#x27;s single package registry is a great idea, for all the reasons that the author notes!

I just used my own docker repo right from the start, precisely to make sure it&#x27;ll be on a domain that I control.<p>In my opinion, every open source project using a foreign-owned domain as their main distribution method is just naive. Of course you&#x27;ll never be able to keep things constant, because you never had any power over that domain.

This is an important conversation and I can&#x27;t help but think that it is repeatedly drawn in predictably valley-minded directions.<p>As a responsible developer your use case is likely: &quot;I want to install package X, and know that my customers are receiving and installing that package when they perform a build&quot;<p>The fact that individuals cannot self-host content has been held back by the limitations of DNS (content addressing) and IPv4 (routability) for a long time now.<p>What I ask is this: if you as a developer were able to self-host the libraries and applications you offer to others -- regardless of whether they&#x27;re open source or proprietary -- would that not solve most of these perverse incentive situations?<p>- The bandwidth costs would be yours, but that would allow you to find a charging model that works<p>- If your software became extremely successful - beyond your own ability to pay for the bandwidth - then the companies and individuals who rely upon your software would be incentivized to step in to foot the bill<p>- Data about software adoption and usage would be de-rigueur shared with the providers who offer the bandwidth for it<p>- We would not be reliant (in a community sense, but also in a day-to-day operations sense) on the benevolent albeit often-loss-leading hosting of centralized repositories

The problem with Docker Hub is that they don&#x27;t let you choose the pricing model. In the end, bandwidth and CPUs are not free, so someone has to pay. For a while it was VCs hoping for growth, but we all know that giving stuff away for free is not sustainable forever.<p>The problem with Docker Hub&#x27;s pricing model is there are actually two use cases for Docker Hub. One is where some random entity makes some software -- they don&#x27;t have any money, so it makes sense for the user to pay to download it. It&#x27;s cheaper than setting up your own CI and hosting to build images, after all. That&#x27;s the only pricing model that Docker Hub supports right now. It may be annoying that it used to be free, but that was just an accident -- you should have been paying for Docker Hub pulls from day one.<p>The other model is where some commercial entity wants to distribute software to their users. In that case, they&#x27;d be happy to pay for their users to anonymously download it. But, Docker Hub doesn&#x27;t support this particular model, and that&#x27;s what&#x27;s causing a lot of problems. (Where I work, this is already a problem for our customers. I think we&#x27;re going to move to GCR, and like the article mentions, this is a pain because we are locked in -- you can&#x27;t make Docker Hub 302 to your new host. Everyone has to rewrite their configuration to pick up the new registry, all because Docker won&#x27;t let us pay for their pulls!)<p>The article also talks about NPM being problematic. I kind of agree with this; I spend a lot of time waiting for my CI system to re-pull node modules. (Because everything is done in containers, nothing is preserved between builds. Things can be preserved if you want them to be, but on CircleCI it&#x27;s actually faster to re-pull all the modules from the Internet than to restore from their own caching mechanism. Shrug.) I had this problem with Go when modules first came out (our builds were getting ratelimited by Github because a lot of our dependencies happened to be hosted there), but it was easy enough to set up a local module proxy, and it was never a problem for us again. (Go has since decided to run their own module proxy, and I stopped running my own. Theirs is great! But obviously, someday it won&#x27;t be free anymore, but at least running your own is a first-class option.) NPM, it seems, have never really offered this as an option -- they&#x27;ve never rate limited me, and their documentation says &quot;don&#x27;t worry about it!&quot; which of course makes me worry more ;)

This is a common pattern how to break pricing mechanism and prevent competition - separate choosers of the service from payers of the service. In this case free hosting is offered to publishers, they choose to host their images there and then users who download them are asked to pay. Had publishers been asked to pay, they would select appropriate registry and perhaps self-host one rather than to use third-party one.

I use a number of dependencies.<p>Most, I wrote, myself, maintain on GitHub, and include as Swift Package Manager dependencies. I write in a modular, layered fashion, and try to fork off as many components as possible into standalone projects.<p>I&#x27;m very, very careful about including third-party dependencies. I think that these are the only ones that I use, throughout my projects:<p><pre><code> SOAPEngine (Paid) ffmpeg VLCLib SwiftKeychain </code></pre> The first, I downloaded and installed directly into my repo (no live link), the two video libs, I use Carthage to include from their home repos, and the last, SPM (also from the home repo). No real registries. I am not a fan of CocoaPods. I use Homebrew for some dev utils on my computer, but the above list is what ships.<p>I may have one more, somewhere, but I can&#x27;t remember, and I&#x27;m too lazy to look. We can rest assured that it was not lightly added.

We have IPFS and the code for hosting most registries is open-source. If the opensource community really wanted to &#x2F; got annoyed enough, it would devise a system that used those components to make a distributed package registry.<p>It&#x27;s easy to complain, it&#x27;s more difficult to work on solutions. We should all be doing more of the latter (working on solutions).

Relying on generosity when there are infrastructure costs does not seem to be a workable model. It&#x27;s somewhat amazing that it has survived so long.

Question: What keeps an organization from hosting it&#x27;s own package mirror internally and only periodically fetch the diffs from the central registry?

Maybe a content-addressable P2P web where everyone has their own cache would be better as a way of distributing packages. It would spare a lot of bandwidth costs for the hosts and maybe make us less dependant on big corporate benefactors.

The Golang package dependency story has been bumpy, but one thing they got absolutely right is not having a package registry.<p>Anyone can host their golang package at example.com&#x2F;myPkg and Go wouldn&#x27;t care less what&#x2F;who runs example.com

I don’t know the specifics about Docker, but at least for NPM the article got some of the things wrong:<p>&gt; ..npm, and other comparable registries are incentivized to create lock-in<p>The author’s conclusion doesn’t match the actions of NPM. There are open source implementations of both client and server, you can run your own NPM registry, and you can change the default registry easily.<p>&gt; What makes npm’s particular scenario even worse is that they&#x27;ve made it so difficult to use a registry that is not npm<p>Uh?! I’m using Verdaccio (an open source Npm registry) everyday, the setup is extremely easy. Also yarn uses their own registry by default.<p>I don’t see where is the “lock in” in NPM.<p>Edit: typo

What about self hosting? Anything but the most popular projects should be possible to host on a Raspberry Pi with static web pages and torrents for large downloads. Off your own home network...

For anyone interested in opting out of scarf&#x27;s analytics when installing NPM packages in any of your environments:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;scarf-sh&#x2F;scarf-js#as-a-user-of-a-package-using-scarf-js-how-can-i-opt-out-of-analytics" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;scarf-sh&#x2F;scarf-js#as-a-user-of-a-package-...</a>

It would be wonderful to keep the service free, but it&#x27;s also a fair point that the data usage must be tremendous.<p>Perhaps a balance could be achieved by rate limiting by IP to encourage caching by devs. I&#x27;m guessing a fair amount of waste occurs from CI testing. Normally once an end user has the image that should be enough.

It works well with youtube. Why not do the same for software? Advertisement is obviously not an option but one could easily ask companies to pay a few k$ a year for professional access. Then just take 30% or so as the platform and redistribute the rest among the uploaders.

Yes, finding a place where you can cut off someone&#x27;s air supply is a good business model.

And why not, a model where unsecure packages are not accessible anymore.

And that&#x27;s what you get for using corporate-drive open source.

Of course you <i>COULD</i> just pay them a reasonable fee for their service.

Another &quot;I want free shit, I want it to be unlimited, have perfect quality, and I want it yesterday&quot; whine.