Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

As someone who spent a good part of my professional career in forecasting and time-series analysis I would like to point out that &quot;point-forecasts&quot; are mostly useless in many import practical applications such as FinTech, e-commerce, sports betting, etc. Point-forecast models such as Prophet fail to give you a meaningful measure of uncertainty of the predicted value. A much better approach are probabilistic forecasting models that predict the probability distribution of the random variable of interest [0]. Probability distribution is the right language to express prediction values and their uncertainty at the same time. And the decisions based on forecasts oftentimes take the uncertainty measures in consideration, e.g. portfolio risk optimization or buying decisions in e-commerce.<p>[0] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Probabilistic_forecasting" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Probabilistic_forecasting</a>

I work with highly seasonal data (city-wide water consumption) and Prophet has been a great tool for us.<p>In terms of performance, it has been the best for a few of our forecasts, compared to GRUs, LSTM, ARIMA and SARIMA. When it wasn&#x27;t the best, it wasn&#x27;t too far from the best model. But, to be fair, our forecast are of quite stable data, so most models do well.<p>However, I would say that the key strength of Prophet is how easy it is. You can produce results really fast, you can throw data with missing range, holidays, and it has interpretability components out of the box. It depends on what do you need, but for most of our tasks, we and our stakeholders are more than happy to sacrifice a bit of performance for this features.

Of course Prophet is a Python library, thus the Monty Python&#x27;s Life of Brian reference:<p>&quot;Now you listen here. He&#x27;s not the messiah. He&#x27;s a very naughty boy. Now GO AWAY!&quot;<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=3_kKAeh6qyc&amp;ab_channel=dingerbell" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=3_kKAeh6qyc&amp;ab_channel=dinge...</a><p>&quot;Yes, we are all individuals!&quot; &quot;Yes, we are all different!&quot; &quot;I&#x27;m not.&quot;:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=KHbzSif78qQ&amp;ab_channel=lucasbeer" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=KHbzSif78qQ&amp;ab_channel=lucas...</a><p>Prophets:<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=hmyuE0NpNgE&amp;ab_channel=radiac" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=hmyuE0NpNgE&amp;ab_channel=radia...</a><p>Life of Brian:<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Monty_Python%27s_Life_of_Brian" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Monty_Python%27s_Life_of_Brian</a><p>Not the Messiah (He&#x27;s a Very Naughty Boy):<p><a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Not_the_Messiah_(He%27s_a_Very_Naughty_Boy)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Not_the_Messiah_(He%27s_a_Very...</a>

My experience in general is that most time series model are inadequate in predicting time series except for very trivial cases of seasonalities or simple linear&#x2F;nonlinear trends.<p>I think that you can throw any model you like at the problem but all you will do is overfit most of the time.

There&#x27;s a reaction from one of the library&#x27;s authors <a href="https:&#x2F;&#x2F;twitter.com&#x2F;seanjtaylor&#x2F;status&#x2F;1410447403153457152" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;seanjtaylor&#x2F;status&#x2F;1410447403153457152</a> <a href="https:&#x2F;&#x2F;threadreaderapp.com&#x2F;thread&#x2F;1410447403153457152.html" rel="nofollow">https:&#x2F;&#x2F;threadreaderapp.com&#x2F;thread&#x2F;1410447403153457152.html</a> .<p>(IDK anything about this subject, just sharing the link.)

I&#x27;m the author. AMA. TMA. I&#x27;m genuinely interested in understanding the popularity of prophet, which as I point out is a non-trivial statistical endeavor.<p><a href="https:&#x2F;&#x2F;microprediction.github.io&#x2F;timeseries-elo-ratings&#x2F;html_leaderboards&#x2F;univariate-k_003.html" rel="nofollow">https:&#x2F;&#x2F;microprediction.github.io&#x2F;timeseries-elo-ratings&#x2F;htm...</a>

Facebook also recently put out a toolkit for time series analysis called Kats<p><a href="https:&#x2F;&#x2F;facebookresearch.github.io&#x2F;Kats&#x2F;" rel="nofollow">https:&#x2F;&#x2F;facebookresearch.github.io&#x2F;Kats&#x2F;</a>

Prophet is not my favorite time series forecasting package. I agree with the author’s findings.<p>Microprediction.com (the site that wrote this article) is a great place to win monthly cash prizes for actually providing forecast accuracy, the twist is you have to provide your forecast as a sample of 225 points from a forecast distribution rather than just providing a point forecast. This makes participation “interesting”.<p>Also to win you have to be more accurate than everything and everyone else that is providing predictions.

i&#x27;ve used prophet along many other ts methods for price forecasting in energy trading. my experience is that prophet is ok, but rather opaque. having tried many packages, I&#x27;ve always come back to the classical statistical methods seeing benefits in transparency (what&#x27;s actually going on? impact of regressors?), speed and most importantly that these methods force the user to think about what&#x27;s happening in the data and make conscious decisions about how to model things. but i can see that sometimes you dont care too much about accuracy and understanding but that you just want a forecast for something that works decently without much hassle.

I&#x27;ll try to give my perspective, though it&#x27;s mostly expanding on em500&#x27;s [1] 3rd paragraph. I think you&#x27;re asking the wrong question about prophet - most users don&#x27;t care &quot;how good is it?&quot; but instead &quot;is it good enough?&quot;, and then &quot;how easy is it to use?&quot;<p>Prophet solves a broad class of easy problems that a lot of ordinary businesses have: you have several years of basically regular data (sales or page views or store foot traffic) that you <i>know</i> has yearly&#x2F;weekly&#x2F;daily (if you have sub-daily data) cycles, and you want to give a reasonable prediction to the business so they can plan for the upcoming week&#x2F;month&#x2F;year. And you want to remove the periodic effects so you can see the underlying trends.<p>Imagine someone, lets call them Bill, who might be called a data scientist, or business analyst or just assistant operations manager, for a medium-large business. Bill has the last 5 years of sales&#x2F;views&#x2F;traffic data in the database (anything before that is in a bunch of excel spreadsheets on the share drive), and knows just enough python to be dangerous. Bill can probably explain an R-squared value but is not an expert at statistics by any measure. He wants to fit the data, but has several problems:<p>1) the weekly trend does not line up with the yearly data, as the year starts on a different weekday.<p>2) Those damn public holidays, some of them occur on a specific date, some of them on the &quot;first Monday of the month&quot;, and some of them seem to change almost randomly year-to-year.<p>3) The reporting system was down for a couple of weeks in June and Feb last year, and the numbers for the first few years were copied from excel, so sometimes are missing the first or last day of the month.<p>Prophet comes by default with yearly&#x2F;weekly seasonality. Prophet comes out-of-the-box with a simple way to import holidays, and even a way to specify your own. Prophet doesn&#x27;t require any cleaning, or special procedures to deal with missing data. And it is quick and easy to use, and get nice-looking, broadly reasonable graphs out (with the above mentioned, consistent data). And that solves the business problem.<p>And Bill&#x27;s probably heard of it because it is (a) popular already, and (b) has Facebook&#x27;s name attached.<p>That&#x27;s my take as to why, even if it is not even close to the most accurate method, Prophet is so broadly popular.<p>[1] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27697274" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27697274</a>

Like most models its data dependent. Had quite a lot of success (was paid) using it on data with multi-seasonality (daily plus seasonal trend) with regressors and change points, where there is not a lot of other options.<p>As its a General Additive Model you can decompose the prediction into parts put them in front of a non-technical user for validation i.e. show effect of daily seasonality, yearly, holidays and regressors. You could even use it to show visually where the model is going wrong for predictions on a blog post ;)<p>Is it the most accurate model on all time series? No but it is useful and good enough for certain use cases.<p>I find it quite interesting what you can do with about 100 lines of stan code. Here is good link on some one building prophet in pymc3 rather than stan to explain its innards.<p><a href="https:&#x2F;&#x2F;www.ritchievink.com&#x2F;blog&#x2F;2018&#x2F;10&#x2F;09&#x2F;build-facebooks-prophet-in-pymc3-bayesian-time-series-analyis-with-generalized-additive-models&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.ritchievink.com&#x2F;blog&#x2F;2018&#x2F;10&#x2F;09&#x2F;build-facebooks-...</a><p>If you want something more flexible you can drop down to this level of code i.e. pymc3, pyro, tfp and bsts. If just want a univarate forecast then ensembles of state space methods are hard to beat as evidenced by the M competitions.<p>But It’s Tough to Make Predictions, Especially About the Future

I&#x27;m a professional forecaster (i.e. getting paid for it) at a large e-commerce company. We have extensive experience with Prophet and a host of other approaches (all the traditional models in Hyndman&#x27;s book&#x2F;R package, some scattered LSTM&#x2F;NN implementations). Here&#x27;s my quick take (the article is a lot more extensive than the median blogpost, and likely warrants a more extensive study than I have time for right now.)<p>Prophet main claims (&quot;Get a reasonable forecast on messy data with no manual effort. Prophet is robust to outliers, missing data, and dramatic changes in your time series.&quot;) are surely exaggerated. As the article shows, time series come in many different shapes, and many of them are not handled properly. It deals well with distant-past or middle-of-the-sample outliers, but not with recent outliers. It cannot deal with level changes (as opposed to trend&#x2F;slope changes). None of this should be a surprise if you take some time to understand the underlying model, which unlike most neural nets is very easily to completely understand and visualise: it&#x27;s really a linear regression model with fixed-frequency periodic components (for yearly seasonality and weekly seasonality) and a somewhat-flexible piecewise-linear trend. The strong assumption that the trend is continuous (with flexible slopes that pivot around a grid of trend breakpoints, which are trimmed by regularisation) accounts for most of the cases where the forecasts are clearly wrong.<p>That said, it does occupy a bit of a sweet spot in commercial forecasting applications. It it&#x27;s largely tuned for a few years of daily data with strong and regular weekly and yearly seasonalities (and known holidays), or a few weeks&#x2F;months of intraday and weekday seasonalities. Such series are abundant in commerce, but a bit of a weak spot for the traditional ARIMA and seasonal exponential smoothers in Hyndman&#x27;s R package. These tended to be tuned on monthly or quarterly data, where Prophet often performs worse. In our experience, for multiple years of daily commercial-activity data, there are no automated approaches that easily outperform Prophet. You can get pretty similar (or slightly better) results with Hyndman&#x27;s TBATS model if you choose the periodicities properly (not surprising, as the underlying trend-season-weekday model is pretty similar as Prophet, but a bit more sophisticated). Some easy win for the Prophet devs are probably to incorporate a Box-Cox step in the model, and a sort-term ARMA error correction, then the model really resembles TBATS. You can usually get better results with NNs that are a bit more tuned to the dataset. But if you know nothing a priori about the data except that it&#x27;s a few years of sales data, your fancy NN will probably resemble Prophet&#x27;s trend-season-weekday model anyway.<p>All of these assume that we&#x27;re trying to forecast any time series&#x27; future only from its own past. If you want to predict (multiple) time series using multiple series as input&#x2F;predictors, that&#x27;s a whole new level of difficulty. I don&#x27;t know of a good automatic&#x2F;fast&#x2F;scalable approach that properly guards against overfitting. Good results for multiple-input forecasting approaches probably requires some amount of non-scalable &quot;domain knowledge&quot;.

This echoes my experience. It often finds seasonality that doesn&#x27;t exist, but it also makes it look like you are doing something impressive...<p>Inexperienced stakeholders don&#x27;t want&#x2F;like to see smooth forecasts, even if you provide predication intervals.

Messiah? No. But one big plus that this article doesn&#x27;t talk much about is how easy it is to get started with it even if you&#x27;re a beginner.<p>A few lines of code gets you a fitted model and some insightful plots. From these you can see if 1) it&#x27;s doing great and you don&#x27;t need to spend hours training some crazy transformer model, 2) It&#x27;s got some flaws and you should maybe try something else (which you can now compare against fbprophet as a baseline) or 3) This data is crazier than I thought, maybe we should rethink things...<p>TLDR: It&#x27;s easy to throw this at a new forecasting problem, and although it isn&#x27;t perfect (as the article shows) sometimes it is still a useful step IMO.

Very entertaining article. That said, as a human, I hard a really had time forecasting most of the time series shown in the article. The ones that had a semblance of regularity it felt like Prophet did a resonable job.

If you are interested in time series predictions, I would suggest you had a look at darts (<a href="https:&#x2F;&#x2F;github.com&#x2F;unit8co&#x2F;darts" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;unit8co&#x2F;darts</a>). It&#x27;s a well designed library which provides a unified API to deal with time series and try&#x2F;compare different algorithms&#x2F;frameworks like Prophet, recurrent NNs, etc.

Their findings are very much in line with my experience with fbprophet, it is usually the least effective lib to predict a time series in my tests.

Can someone ELI5 what Prophet and time series are? Is there ever a chance a layperson would find use in Prophet?

I would recommend Python fans take a look at <a href="https:&#x2F;&#x2F;www.microprediction.com&#x2F;blog&#x2F;popular-timeseries-packages" rel="nofollow">https:&#x2F;&#x2F;www.microprediction.com&#x2F;blog&#x2F;popular-timeseries-pack...</a> and the Elo ratings for timeseries (e.g.) <a href="https:&#x2F;&#x2F;microprediction.github.io&#x2F;timeseries-elo-ratings&#x2F;html_leaderboards&#x2F;univariate-k_003.html" rel="nofollow">https:&#x2F;&#x2F;microprediction.github.io&#x2F;timeseries-elo-ratings&#x2F;htm...</a> The timemachines package wraps prophet and others for the purpose of comparision <a href="https:&#x2F;&#x2F;github.com&#x2F;microprediction&#x2F;timemachines" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;microprediction&#x2F;timemachines</a>

To close, let me say that this post ended up being more negative than I expected and, like Nessie, my opinion may rise in the future when I understand the implications of the Prophet generative model better, and either modify it or find better ways to identify its strengths. The unanswered question here is why Prophet is so popular, and this surely merits a better explanation than I have given. I think there are probably statistical angles I am not seeing - something reflecting the fact that people are voting with their eyeballs when they use Prophet.<p>(from the article)<p>- The author

Is there a way to adapt time-series products like this to analysis of categorical data, in my case audit logs. I&#x27;d love to be able to suss out patterns&#x2F;clusters&#x2F;sequences but most of the stuff I&#x27;ve looked at requires you to do unnatural things like flip the categorical content into distinct dimensions with a calculated rate metric.

Shouldn&#x27;t time-based (weekly and holiday) and trend effects be multiplied rather than added?<p>For instance, if users spend twice as many hours on the weekend on a website, and the total number of users has doubled, then these effects multiply to give 4x visits than the baseline non-weekend.

Lots of good investigative work but its not clear if there is a decidable problem here.

Excellent write up, great to see some attention paid towards libraries built on top of STAN, and answering questions of uncertainty via Bayesian stats.

Need way more data points...

You&#x27;d think people who are hyping AI were trying to bring about the next &quot;AI Winter&quot;.<p>People hear &quot;FAANG&quot; and it suspends their critical judgement.

Naughty boy.

The point of Prophet is to make time series accessible to people who aren&#x27;t experts in time series. Yes, it has some well known failure modes, but using Prophet always beats a static LY &quot;forecast.&quot; This article is like complaining a Nissan Leaf can&#x27;t outrace a Tesla.