This post is a good example of my current issues with modern data journalism as content marketing: you can say <i>anything you want</i> without evidence to back it up, but hey, there's a pretty chart, so it doesn't matter. (In this blog post, the data points <i>don't correspond to actual data!</i>)<p>This is also the primary reason I have switched to Jupyter/R Notebooks for my blog posts: if I make ridiculous claims, people can check my work as evidence. This post doesn't even provide any quantifiable metrics, just "we analyzed millions of commits."
Non-technical co-founder of Git analysis tool writes blog post that makes assertions about all sorts of things and then justifies this with mockup charts whose axes are unlabeled. Then offers tools to help micro-manage engineers.
Neither in the article or in the referenced article/blog post linked to "Impact" is impact defined.<p><a href="https://blog.gitprime.com/impact-a-better-way-to-measure-codebase-change" rel="nofollow">https://blog.gitprime.com/impact-a-better-way-to-measure-cod...</a><p><pre><code> What I found is:
</code></pre>
Impact takes the following into account:<p><pre><code> The amount of code in the change
What percentage of the work is edits to old code
The surface area of the change (think ‘number of edit locations’)
The number of files affected
The severity of changes when old code is modified
How this change compares to others from the project history
</code></pre>
No exact or detailed formula for "impact" is given. "takes the following into account" is extremely vague as it could indicate any relationship. Is impact higher with more files changed in the commit? Why would it be better if more files are changed? Is it lower? Why would it be better if fewer files are changed? Is it some totally unobvious non-linear function such as a trigonometric sine?<p>Based on the vague description above, nothing in "impact" is directly related to the actual end user/paying customer experience or a reasonable proxy such as systematic end user testing by a QA team.<p>This lack of a direct relationship to the desired end result is the same problem that lines of code (loc or LoC) and many other metrics of software engineer output have.<p>The "impact" metric, whatever it precisely is, looks suspiciously like it would naturally be positively correlated with a large number of commits/high frequency of commits.<p>Also the plot is labeled with "volume" on the horizontal axis and not the mysterious "impact" metric. The text implies this horizontal axis is the "impact" metric. Why is the horizontal axis not labeled impact?<p>Even more peculiar, "impact" is claimed to measure cognitive load:<p>Impact attempts to answer the question: “Roughly how much cognitive load did the engineer carry when implementing these changes?”<p>A good engineer will attempt to find a low or no cognitive load solution to a problem! In general this will be faster and less error prone and cheaper! Reinventing the wheel has a very high cognitive load.
The title, well actually the entire article, is one of those things where A -> B is stated, well knowing (and almost expecting) that readers will understand it as B -> A or B <-> A. It's the developer version of the typical <i>"10 Things Successful People Do</i>" clickbait.<p>Yes, a lot of (although far from all!) good coders have the habit of taking small bites. That doesn't mean though that by taking smaller bites you are (or are becoming) a good coder. I've had the past displeasure of dealing with code that hit all the superficial checkmarks (small commits, unit-tested, peer-reviewed, you name it...) yet completely stunk.
Jesus that site was annoying. I go to read the article, and a popup beeps at me distracting me. I think "what the hell is this crap" and move my mouse towards the "about" page. A pop-up pops up (as they do) and distracts me from even that.<p>Both times I tried to get information out of their site they found a way to interrupt me.<p>Then I leave without reading the article.
I large agree with this, although with some provisos.<p>First of all, sometimes it's most useful to take a step back and think about problems in a wider context. Often the biggest impact is from the code you don't write. There's really no way those kinds of contributions come out in any kind of metrics, but the team will sure remember them.<p>More commonly, I prefer to commit often to aid my dev process but then rebase those commits into more cohesive units before merging. How chunky to make those final commits is somewhat a matter of taste, and probably shaped by the domain one is working in. Personally I like the finest grain possible without any build breakage, that way git-bisect still works, but hopefully there's still enough granularity to help with trickier merge/rebase scenarios around dependency upgrades and/or db migrations. But pardoning the digression, my point is the output seen by the rest of the team may not represent the original frequency of commits.
Title of the article uses the word "Engineers" rather than software engineers, and if you lump together virtually anybody who makes changes to a code base (testers, sw architects, sw leads, etc), it seems like its easy to have the data obviously skewed by people who essentially spam the commit process. I think most people can look at a commit log and sort of eye-ball who are the people who really make shit happen on a project--and who is probably causing more grief than anybody else. I agree its hard to quantify, but to be honest I didn't take away much from this article. I like the goal of the article, just think it didn't really hit the mark.
Extrapolating a "prolific engineer" from one's commit log is no less absurd than from lines of code.<p>Their methodology fails to account for so many things, commit squashing feature branches, for example. A fundamentally flawed "study".
What are they measuring? Public git repositories? The article doesn't say anything about whatever data is being analyzed. If its public git that's hardly comparable to a large corporate entity.
Analyzing churn is tricky and can be easily misinterpreted by management. This reminds me of what was revealed from studies in behavioral sciences and presented by Dan Ariely: <a href="https://www.youtube.com/watch?v=Q92BqouxyX4" rel="nofollow">https://www.youtube.com/watch?v=Q92BqouxyX4</a>
Can you make any claims about the number of commits? That's another metric that changes depending on tools and processes.<p>If you're using a code review tool well (if you are more experienced with it) then you'll automatically be making lots of small commits vs people that may not be well versed in working that way.<p>If you're working with large or prolific teams then you'll be committing all the time with eg feature toggles to ensure everyone is as close to the current code with their changes as possible.<p>So making general claims about more commits without taking into consideration the ways that metric is bent and changes with tools and processes and different experiences seems to be dangerous.
Unless I'm missing something, how is 'Impact' derived here?<p>Impact as a metric is fine, but you have to show what it is and how you arrived at that conclusion, and have verified that against a test set of data otherwise it's turtles all the way down.<p><a href="https://en.m.wikipedia.org/wiki/Turtles_all_the_way_down" rel="nofollow">https://en.m.wikipedia.org/wiki/Turtles_all_the_way_down</a>
We have a hard rule that you do not commit code that doesn't pass unit tests.<p>This leads to bad coders not being able to commit as frequently as good coders.<p>However, this seems like a very, very loose heuristic.
concepts missing from the article: 'staging' and 'commit size'.<p>People who have worked in code review shops tend to stage their commits, i.e. do a bunch of work but commit it with git commit -p.<p>Article also doesn't look at deployment frequency, and 'merge' appears only once. We don't know if these 'high impact' devs are in their own branch for 6 weeks cooking up the PR from hell. That's one way to have a high impact.
I'm curious what the general vehemence is to quantifying and understanding this information is? It's right there in your commit history. I've run my own analyses out of curiosity in the past... statistics just isn't my favourite subject. I think it's an interesting metric and I'd certainly like to know more about how it's calculated.<p>(Although I'm sure that's part of the secret sauce).
This reminds me that one of the things I loved about working on server-side efficiency is that for any given diff, you can measure the performance improvement, multiply by hardware costs, and say, "That diff saved $X." Makes it really easy to measure impact and feel that my salary is justified.
This is misleading, however I would like to point out that if Small Bites == Kaizen. Then, Prolific Engineers like any other prolific professionals may be Kizening the crap out of their codebases. No data here either to back the smart ones up though.
Unfortunately know some bottom-right quadrant people. We got rid of some of those not too long ago - wouldn't want them on my team again.
Super hard to manage and work with.
There's a great, fun, programmer-centric website called thedailywtf.com (sorry, I just scheduled the rest of your afternoon for you). Developers can submit WTF's that they've found in other's code - as the site admin says, "curious perversions in information technology".<p>One thing that strikes me about the majority of the submissions, as funny as they are, is that they mostly boil down to "so-and-so didn't know that such-and-such feature existed, so wrote reams of code to implement that feature in a complex way". It also strikes me that just this article's sort of analysis of "prolific" (aka "good") engineers/programmers drives this same sort of behavior. If every developer is supposed to be committing code all day, every day, there's no time left over to read the product documentation, try out a new feature, review a reference implementation, read a blog post: to be "good", you must be spending as much time as possible _typing_, because that's what you're paid to do. This (ubiquitous) management mentality is how we end up with roll-your-own crypto, or five competing Javascript frameworks, parsing using regular expressions... it's not so much that what they did was wrong - and trust me, if it works, it won't be removed - it's that it's pointless.