Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect.<p>(2) it's not clear what the "error bars" should mean. One is a confidence interval[1] (e.g. model gives 95% chance the output will be within these bounds). Another is a standard deviation (i.e. you are pretty much predicting the squared difference between your own point forecast and the outcome).<p>[1] acknowledged: not the correct term
Yes, please! I was part of an org that ran thousands of online experiments over the course of several years. Having some sort of error bars when comparing the benefit of a new treatment gave a much better understanding.<p>Some thought it clouded the issue. For example, when a new treatment caused a 1% "improvement", but the confidence interval extended from -10% to 10%, it was clear that the experiment didn't tell us how that metric was affected. This makes the decision feel more arbitrary. But that is exactly the point - the decision <i>is</i> arbitrary in that case, and the confidence interval tells us that, allowing us to focus on other trade-offs involved. If the confidence interval is 0.9% to 1.1%, we know that we can be much more confident in the effect.<p>A big problem with this is that meaningful error bars can be extremely difficult to come by in some cases. For example, imagine having something like that for every prediction made by an ML model. I would <i>love</i> to have that, but I'm not aware of any reasonable way to achieve it for most types of models. The same goes for online experiments where a complicated experiment design is required because there isn't a way to do random allocation that results in sufficiently independent cohorts.<p>On a similar note, regularly look at histograms (i.e., statistical distributions) for all important metrics. In one case, we were having speed issues in calls to a large web service. Many calls were completing in < 50 ms, but too many were tripping our 500 ms timeout. At the same time, we had noticed the emergence of two clear peaks in the speed histogram (i.e., it was a multimodal distribution). That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users. That knowledge allowed us to ignore wide swaths of code and spot the speed issues in some recently pushed personalization code that we might not have suspected otherwise.
Completely agree with this idea. And I would add a corollary...date estimates (i.e. deadlines) should also have error bars. After all, a date is a forecast. If a stakeholder asks for a date, they should also specify what kind of error bars they're looking for. A raw date with no estimate of uncertainty is meaningless. And correspondingly, if an engineer is giving a date to some other stakeholder, they should include some kind of uncertainty estimate with it. There's a huge difference between saying that something will be done by X date with 90% confidence versus three nines confidence.
Uncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties.<p>I always demand error bars.
I'm reminded of Walter Lewin's analogous point about measurements from his 8.01 lectures:<p><pre><code> any measurement that you make without any knowledge
of the uncertainty is meaningless
</code></pre>
<a href="https://youtu.be/6htJHmPq0Os" rel="nofollow noreferrer">https://youtu.be/6htJHmPq0Os</a><p>You could say that forecasts are measurements you make about the future.
The interesting example in this article is nowcasting! The art of forecasting the present or past while you're waiting for data to come in.<p>It's sloppy science / statistics to not haven error ranges.
Every estimate/prediction/forecast/interpolation/extrapolation should have a confidence/prediction/ or tolerance interval (application dependent) that incorporates the assumptions that the team is putting into the problem.
Reminds me of this paper[1]<p>> An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability<p>> Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average.<p>[1] <a href="https://www.microsoft.com/en-us/research/publication/an-illusion-of-predictability-in-scientific-results-even-experts-confuse-inferential-uncertainty-and-outcome-variability/" rel="nofollow noreferrer">https://www.microsoft.com/en-us/research/publication/an-illu...</a>
Forecasts can also be useful without error bars. Sometimes all one needs is a point prediction to inform actions.
But sometimes full knowledge of the predictive distribution is helpful or needed to make good decisions.<p>"Point forecasts will always be wrong" - true that for continuous data but if you can predict that some stock will go to 2.01x it's value instead of 2x that's still helpful.
This is a great advantage of Gaussian Process Regression aka. Kriging.<p><a href="https://en.wikipedia.org/wiki/Gaussian_process#Gaussian_process_prediction,_or_Kriging" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Gaussian_process#Gaussian_proc...</a>
Doesn't work.<p>For instance in a business setting, if I say "it'll be done in 10 days +/- 4 days", they'll immediately say "ok so you're saying it'll be done in 14 days tops then".<p>More effective to sound as unsure as possible, disclaim everything in slippery language, and promise to give updates to your predictions as soon as you realise they've changed (granted this wouldn't work as well for an anonymous reader situation like in this article).
Not just forecasts.<p>Accounting should do it too in their reporting.<p>I would love to see a balance sheet with a proper 'certainty range' around the values in there.
For prediction intervals that are guaranteed coverage, check out conformal prediction [1]. Works great especially for time series data.<p>[1] <a href="https://github.com/valeman/awesome-conformal-prediction">https://github.com/valeman/awesome-conformal-prediction</a>
Not only forecast need error bars. Every statistic needs error bars. But even then most people interpret error bars wrongly, see e.g. <a href="https://errorbars.streamlit.app/" rel="nofollow noreferrer">https://errorbars.streamlit.app/</a>
I'm just imagining adding error bars to my schedule forecasting (with schedules that are typically one the optimistic side thanks to management), with bars pointing in the bad direction, and seeing management still insist it'll take too long.
Let me suggest a solution <a href="https://github.com/valeman/awesome-conformal-prediction">https://github.com/valeman/awesome-conformal-prediction</a>
two inches deeper, is too much money!
So Alphabet prefer to loose the idea, the contract, the business plan rather than make the things solid and works for a long time.
a $ is a dollars, a project, is just a piece of fun.
Alphabet LOL XD
I have, in my life as a web developer, had multiple "academics" urgently demand that i remove error bands, bars, notes about outliers, confidence intervals etc from graphics at the last minute so people are not "confused"<p>Its depressing