Counting Objects

This is technical recruiting done right, a lucid walkthrough of a hard problem complete with links to the implementation. It must have taken at least a few weeks of engineer-time to write, Github is awesome for making this public.<p>I wish they had talked a little more about the tradeoff they made. They mentioned that splitting packfiles by fork was space-prohibitive, but ended up with a solution which must take more space than originally used. (If the new heuristic refuses to use some objects as delta bases, some options which would have provided the best compression are no longer available to git)<p>The performance win is incredible, how much space did they give up in the process?

Not specific to the post, but:<p>&gt; we&#x27;re sending few objects, all from the tip of the repository, and these objects will usually be delta&#x27;ed against older objects that won&#x27;t be sent. Therefore, Git tries to find new delta bases for these objects.<p>Why is this the case ? git can send thin packs if the receiver already has the objects, why does it still need to find a full base to diff against ? (Not counting when initial base objects are from another fork -- I don&#x27;t know if it&#x27;s often the case)<p>On top of that as far as I understood from the discussion about heuristics (<a href="https:&#x2F;&#x2F;git.kernel.org&#x2F;cgit&#x2F;git&#x2F;git.git&#x2F;tree&#x2F;Documentation&#x2F;technical&#x2F;pack-heuristics.txt?id=HEAD" rel="nofollow">https:&#x2F;&#x2F;git.kernel.org&#x2F;cgit&#x2F;git&#x2F;git.git&#x2F;tree&#x2F;Documentation&#x2F;t...</a>) it seems like the latest objects are full and the earlier objects are diffed against them (double benefits: you usually want access to the last object which is already full, and earlier objects tend to be only remove stuff, not add because &quot;stuff grows over time). So if objects are still stored as packs, things should already be in a pretty good shape to be sent as-is... or not ?

This is awesome. I absolutely love &lt;company&gt;engineering blogs like this.<p>There&#x27;s no real reason for companies to educate external devs&#x2F;hobbyists&#x2F;students like this, but some do, and it&#x27;s really awesome.

Perhaps I&#x27;m a bit ignorant of git&#x27;s storage and protocols, but what&#x27;s the purpose of this initial count? Seems to me it&#x27;s traversing the tree twice - once to count and once to send the objects across the network. So why not traverse the tree, sending objects that need to be sent and ignoring the ones that don&#x27;t <i>instead</i> of counting?

Hopefully this isn&#x27;t too off topic, does anyone know what software was used to make the charts in this post?

The Azure team responsible for the Local Git implementation needs to read this fantastic article.<p>I&#x27;ve been putting up with 10-minute deploys due to precisely this issue of counting objects. It&#x27;s slow because we don&#x27;t use Local Git as our source-of-record repository (because commits initiate a deployment step), so every deploy involves a clean fetch into a new tmpdir.<p>At least now I know why our deploys are getting slower and slower.

I wonder why GitHub has a separate domain githubengineering.com for this blog instead of a subdomain like engineering.github.com.<p>I notice that there is an inactive user account called engineering. If at all an User Page is created by that account, it would be available at engineering.github.io.

So does GitHub hire C engineers? Git is implemented in C.

The illustrations look like they were done in Paper. Does anyone know if that&#x27;s right or not?

Could this technique be used to optimise graph traversal in a postgresql table? Or are recursive queries still the way to go?

Not only is the tech achievement amazing, and something that will save countless man hours it&#x27;s incredibly written to boot.<p>Amazing work.

&quot;When you fork a repository on GitHub, we create a shallow copy of it. This copy has no objects of its own, but it has access to all the objects of an alternate ...&quot;<p>So does this mean one could attach a GitHub repository by having a lot of shill accounts cline it and add random objects (possibly having a performance impact on the original)? I understand the engineering need for the use of alternates, but wonder about the lowered degree of isolation.