GitHub’s Large File Storage is no panacea for Open Source

In the interest of not propagating this common misconception:<p>&quot;The main problem with Git is that binary files are stored “as is” in the history of the project, so that every single revision of a new binary file (even if just a single byte has changed) is stored in full. [...] On the other hand, source files being mostly text, they are more intelligently handled and typically only differences between revisions are stored in the commits.&quot;<p>This is false. Git stores the full version of each file in &quot;loose&quot; format and uses compressed incremental diffs (originally based on xdiff) in packfiles (after &quot;git gc&quot;) without distinguishing text vs binary in either case. The issue is that binary files are often compressed themselves (so a one-byte semantic change has nonlocal effect) or have positional references (like jump targets in an executable, causing small changes to cascade).<p>These factors explain the inefficient handling of binary files, but improving efficiency requires changing the semantics. LFS follows in the path of a few other tools (based on smudge&#x2F;clean filters) that try to hide the semantic difference from the casual user, though that difference seems to bite people more frequently than we&#x27;d like.

How uncharitable can a single blog post be! The entire post is discredited by the author repeatedly projecting his unfounded opinions onto GitHub, such as<p>&quot;My guess is that some high-level greedy marketing dickwad, completely unaware of the asinine implications of his brilliant idea, signed off on this dumb-as-a-bag-of-rocks pricing model.&quot;<p>&quot;All the marketing material pimping GitHub’s LFS support [...]. I do not believe this is unintentional.&quot;<p>&quot;This is completely batshit. The side effect of this pernicious, greedy pricing model is to [...]&quot;<p>&quot;I honestly couldn’t believe that GitHub would be willing to do something that shortsighted, visibly motivated by greed from the cash they thought they could extract from some of their users&quot;.<p>Charitable explanation for forks not working: they haven&#x27;t yet written the code to make this work with forks, and it&#x27;s better to ship something working early, than to make it work in all cases.<p>Charitable explanation for charging for bandwith: bandwidth costs money. (I believe this is a real problem for Dropbox, which doesn&#x27;t charge for bandwidth but must still pay for it). Also, all CDNs, and also AWS charge for bandwidth.<p>Overall, while GitHub may be able to support it&#x27;s OSS folks better by changing the pricing on some parts of its product, this post is incredibly uncharitable. I hope the OP will consider removing the unfounded narrative that he&#x27;s projecting onto GitHub (esp the &quot;marketing dickwad&quot; thing - wtf) and focus on the facts.<p>[Disclaimer: my company partners with GitHub on lots of stuff]

&gt; Case in point: if a very popular Github repository (such as the one for the Linux kernel) decided to start using LFS for some of their files, they would instantly alienate all of their users. They would no longer be able to properly fork the project, or even clone it to get its binary files stored via LFS. Nobody would be able to send a pull request to Linus as a result without considerable effort.<p>Odd example. Linux doesn&#x27;t use GitHub pull requests.

There&#x27;s a lot of assumptions here about GitHub being greedy. I&#x27;ve got no idea how much money it costs GitHub to support Open Source projects, but it must easily be in the millions. I think that by this point GitHub deserves the benefit of the doubt before launching into vicious accusations.

The most interesting takeaway for me was that Microsoft seems to privide the only(?) free git hosting that includes LFS?<p>Does anyone know if their repos supports forking in combination with LFS too?

This seems like an odd problem, but I&#x27;m not as familiar with Git as I should be. Is there not a reasonable way to download only the most recent version of these large binary files on the initial request, and then download the historical versions only in the (likely very rare) case that the user actually wants to use them? This would seem more useful in this case than hoping that binary diffs the repository small enough.

Edit: I was wrong, however I learned from the conversation so I am leaving it here! Thanks to those who corrected me.<p>&gt; On the other hand, source files being mostly text, they are more intelligently handled and typically only differences between revisions are stored in the commits.<p>This is completely incorrect, git stores whole blobs from one commit to the other.<p>svn stored patches, but git does not. Every version of a file is stored in its entirety in your git tree since the beginning of the repository&#x27;s existence. This is one of the reasons why git is so fast. You can go through your objects in your .git directory and verify this for yourself[0].<p><pre><code> $ find .git&#x2F;objects -type f .git&#x2F;objects&#x2F;ff&#x2F;a5d733354ae6f8bdc67764d58d87c9a3161f66 .git&#x2F;objects&#x2F;ff&#x2F;deb08f4856bd6eb5b31d7f800b3e480ae3e2e0 $ git cat-file -p ffa5d733354ae6f8bdc67764d58d87c9a3161f66 ...file contents appear... </code></pre> [0] <a href="https:&#x2F;&#x2F;git-scm.com&#x2F;book&#x2F;en&#x2F;v2&#x2F;Git-Internals-Git-Objects" rel="nofollow">https:&#x2F;&#x2F;git-scm.com&#x2F;book&#x2F;en&#x2F;v2&#x2F;Git-Internals-Git-Objects</a>

I suspect setting up the free LFS reference&#x2F;test server[1] that GitHub provides would have taken less time than writing this post complaining that GitHub isn&#x27;t free enough.<p>1: <a href="https:&#x2F;&#x2F;github.com&#x2F;github&#x2F;lfs-test-server" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;github&#x2F;lfs-test-server</a>

At GitLab we&#x27;re working to support LFS. Initial support might or might not work with forks. As now with our Git Annex support storage will be free with a soft limit of 10GB of disk space per project (includes Git, Git Annex and Git LFS data) and there is no bandwidth limit. It will work with public and private projects (both are free).

I&#x27;m not sure I understand why artifacts can&#x27;t be stored in a different service - even an S3 bucket, if not a real repository service - and fetched dynamically via a build process.<p>Is there a reason why binary blobs need to be stored directly next to code in order to be versioned?

I wonder if Perforce Cloud will be able to fill this role at all. Probably not. Open Source isn&#x27;t their target audience. But it could be a consideration.<p>Has anyone tried the new Perforce&#x2F;Git stuff? Is it any good? We&#x27;re still on an older pre-Helix version.

The post seems hyperbolic. I&#x27;d love to hear GitHub&#x27;s rebuttal.

&gt; I honestly couldn’t believe that GitHub would be willing to do something that shortsighted, visibly motivated by greed from the cash they thought they could extract from some of their users.<p>They&#x27;re a business. C&#x27;mon here.

Shock and horror: commercial company has a paid value add.<p>GitHub is not a charity.