There is a very good essay in the first comment by "Roger" dated Jan-2023, reproduced below. Skip the primary essay and work from this:<p>"I really appreciated this piece, as designing good metrics is a problem I think about in my day job a lot. My approach to thinking about this is similar in a lot of ways, but my thought process for getting there is different enough that I wanted to throw it out there as food for thought.<p>One school of thought 9<a href="https://www.simplilearn.com/tutorials/itil-tutorial/measurement-in-itil-malc" rel="nofollow">https://www.simplilearn.com/tutorials/itil-tutorial/measurem...</a>) I have trained in is that metrics are useful to people in 4 ways:<p><pre><code> 1. Direct activities to achieve goals
2. Intervene in trends that are having negative impacts
3. Justify that a particular course of action is warranted
4. Validate that a decision that was made was warranted
</code></pre>
My interpretation of Goodhart’s Law has always centered more around duration of metrics for these purposes. The chief warning is that regardless of the metric used, sooner or later it will become useless as a decision aid. I often work with people who think about metrics as a “do it right the first time, so you won’t have to ever worry about it again”. This is the wrong mentality, and Goodhart’s Law is a useful way to reach many folks with this mindset.<p>The implication is that the goal is not to find the “right” metrics, but to instead find the most useful metrics to support the decisions that are most critical at the moment. After all, once you pick a metric, 1 of 3 things will happen:<p><pre><code> 1. The metric will improve until it reaches a point where you are not improving it anymore, at which point it provides no more new information.
2. The metric doesn’t improve at all, which means you’ve picked something you aren’t capable of influencing and is therefore useless.
3. The metric gets worse, which means there is feedback that swamps whatever you are doing to improve it.
</code></pre>
Thus, if we are using metrics to improve decision making, we’re always going to need to replace metrics with new ones relevant to our goals. If we are going to have to do that anyway, we might as well be regularly assessing our metrics for ones that serve our purposes more effectively. Thus, a regular cadence of reviewing the metrics used, deprecating ones that are no longer useful, and introducing new metrics that are relevant to the decisions now at hand, is crucial for ongoing success.<p>One other important point to make is that for many people, the purpose of metrics is not to make things better. It is instead to show that they are doing a good job and that to persuade others to do what they want. Metrics that show this are useful, and those that don’t are not. In this case, of course, a metric may indeed be useful “forever” if it serves these ends. The implication is that some level of psychological safety is needed for metric use to be more aligned with supporting the mission and less aligned with making people look good."