> And in any case, the model only generated probabilities of winning a game and advancing, and no team was given more than an 18.5 percent chance of winning the World Cup.<p>> [...]<p>> But Goldman Sach’s misfire is perhaps the most curious.<p>The model said, that there is a lot of uncertainty, and as it happens, it was entirely correct. A World Cup chance of 18.5 percent means, that 4 out of 5 times the team will not win, and that that is the highest chance does not say much about the model.<p>And in general this is one instance of the well practiced journalistic technique to wait for results first and then define a bar afterwards to criticize the results according to standards that did not exist when the performance happened. (I guess in this case it is even worse, we could construct a reasonable test of the model performed, I have the suspicion that that was in the original paper and that the journalist either did not understand it, or, more likely, choose to ignore it in favor of writing a better story.)
Leonid Bershidsky and a lot of other journalists laughing at Goldman Sachs' incorrect predictions seem to miss the point.<p>The World Cup predictions from Goldman Sachs (and also UBS) are a form of <i>recreation and entertainment</i> with machine learning. It's an expression of quant nerd humor.<p>Analogous intellectual games would be engineers devising ridiculous Rube Goldberg contraptions[1] or programmers building "enterprise" FizzBuzz[2].<p>(I think it would add to the fun if GS uploaded their raw data and models to Github for others to play with.)<p><i>>It certainly didn't predict the final opposing France and Croatia on Sunday.</i><p>True, but it did predict France having better chance winning overall but was handicapped by a tougher draw. It also predicted France beating Croatia in round 16 instead of the final. The pdf says:<p><i>>While Germany is more likely to get to the final, France has a marginally higher overall chance of winning the tournament, </i><p>[1] <a href="https://en.wikipedia.org/wiki/Rube_Goldberg_Machine_Contest#Past_tasks" rel="nofollow">https://en.wikipedia.org/wiki/Rube_Goldberg_Machine_Contest#...</a><p>[2] <a href="https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition" rel="nofollow">https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...</a>
People conflate statistics with actual results more often than not and I think those reporting on such stories and maybe even the original authors might fall for this.<p>It was not wrong to say Hillary had a 95% chance of winning the presidential election, but the confidence was low and that value <i>still allowed for the opposite result to happen</i>.<p>Also football has a lot of variance concerning team capability and end results. The better team might (and does) lose often, especially when going to penalty shoots.<p>With basketball, the stronger team will be easily scoring more in most cases.
The World Cup is about the worst sporting event for data led predictions like this, far too much can rely on a few events that are basically a coin flip. It would be interesting to see how the predictions went for something like the Premiere League tables.
People love to beat up on these companies because of this stupid world cup prediction. Yes, Goldman is a giant vampire squid wrapped around the face of humanity (Matt Taibi quote). But it turns out it's really just great marketing for their research teams.<p>Also, I've seen some people say (not in this forum) that banks now look stupid because they're in the business of making predictions and they can't even get the world cup right. Guess what? Banks make no money on predictions. They make money on flows and taking spreads on trades they do with clients. Any research or prediction is meant to be a catalyst for that trade.
I am somewhat shocked that GS would jump into the prediction business of the World Cup, even as joke. The risk of people getting the wrong idea about the prediction and GS itself is too great, even with a perfectly defensible model.<p>This is an enterprise for bookies, not Goldman Sachs.
The "Ludic Fallacy" strikes again [0].<p>> The ludic fallacy, identified by Nassim Nicholas Taleb in his 2007 book The Black Swan, is "the misuse of games to model real-life situations."<p>...<p>> The alleged fallacy is a central argument in the book and a rebuttal of the predictive mathematical models used to predict the future – as well as an attack on the idea of applying naïve and simplified statistical models in complex domains. According to Taleb, statistics is applicable only in some domains, for instance casinos in which the odds are visible and defined.<p>Both Taleb's books, "The Black Swan" and "Fooled by Randomness" are an interesting take for such models. Meanwhile, most economists know about "Knightian Uncertainty" [1] which talks about differentiation of risk and uncertainty.<p>> "Uncertainty must be taken in a sense radically distinct from the familiar notion of Risk, from which it has never been properly separated.... The essential fact is that 'risk' means in some cases a quantity susceptible of measurement, while at other times it is something distinctly not of this character; and there are far-reaching and crucial differences in the bearings of the phenomena depending on which of the two is really present and operating.... It will appear that a measurable uncertainty, or 'risk' proper, as we shall use the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all."<p>[0] <a href="https://en.wikipedia.org/wiki/Ludic_fallacy" rel="nofollow">https://en.wikipedia.org/wiki/Ludic_fallacy</a><p>[1] <a href="https://en.wikipedia.org/wiki/Knightian_uncertainty" rel="nofollow">https://en.wikipedia.org/wiki/Knightian_uncertainty</a>
If anything, this is a clear illustration of poor use of probabilistic prediction. When used for investments you have many outcomes. If the model is any good, you will most of them right. In the World Cup you have very few. Even if you count all games played. Definitely not excusing Goldman Sachs here, they should have known better than to try to predict this. There was only a tiny chance this could be great advertisement for their model.
PS: If you want to build or train your own model or make predications, you can find open (structured) data about all world cups at the football.db, see <a href="https://github.com/openfootball/world-cup" rel="nofollow">https://github.com/openfootball/world-cup</a> and <a href="https://github.com/openfootball/world-cup.json" rel="nofollow">https://github.com/openfootball/world-cup.json</a> Enjoy the beautiful game.
The predictions were not <i>so</i> bad. At least one of the favourites won in the end. GS had France winning with 11.3% probability, second to Brazil with 18.5%. UBS was less fortunate, they had Germany (24%), Brazil (19.8%), Spain (16.1%) and England (8.5%) before France (7.3%).<p>I compared the logloss for their predictions with the "uniform" benchmark (giving each team 1/32 probability of winning, 1/16 probability of getting to the finals, etc) and the results are the following (if I transcribed the data properly):<p>Getting to second round:<p>GS: 0.495
UBS: 0.495
bench: 0.693<p>Getting to quarter-finals:<p>GS: 0.463
UBS: 0.459
bench: 0.562<p>Getting to semi-finals:<p>GS: 0.310
UBS: 0.327
bench: 0.377<p>Getting to final:<p>GS: 0.231
UBS: 0.269
bench: 0.234<p>World-cap winner:<p>GS: 0.097
UBS: 0.113
bench: 0.139<p>The performance of the models was ok until Croatia got to the finals. This hurt specially UBS, who predicted less than 0.9% probability of such an event (compared to 2.1% in Goldman's model).<p>Edit: these would have been the "best case" scores (if the high-probabilty teams had classified to each round, ignoring that this may be impossible due to the structure of the tournament):<p>GS: 0.432 0.302 0.220 0.141 0.079<p>UBS: 0.365 0.251 0.176 0.111 0.070<p>UBS could potentially achive lower logloss metrics because it had more extreme predictions.
Isn’t this a little like flipping a coin four times - getting heads four times in a row, and looking at your friend and saying “but you told me the odds were 50/50 each flip?!”
This article didn't compare the Goldman Sachs model to any other models-- why not compare it with sports betting odds? Would Goldman have made or lost money betting their model was better than the crowd?
>Soccer, with the many factors that affect game outcomes — players’ injuries and intra-team conflicts, the refereeing, the weather, coaches’ errors and moments of inspiration — remains only a tightly-regulated game involving a <i>few dozen people</i>. The behavior and performance of big corporations, <i>entire industries and nations</i> is arguably even more difficult to model based on data about the past.<p>Author misses the way models work entirely, the larger the entity, the more statistics and averages kick in, and as a result, better model can be built.
I watched quite a few matches and among the things I saw in the matches but not in any statistics are:<p>- motivation (Germany and Croatia were the two extremes here, no idea how to measure it)<p>- team cohesion (number of articles in a few journals questioning the team cohesion, maybe also articles about individual players)<p>- creativity in offense (maybe measurable via "target missed from close distance" + "ball passed front of the goal")<p>- number of errors in defense that didn't lead to a goal<p>- percentage of times ball possession was lost from own goal to enemy's area (England was really bad here against Croatia)
<i>Thanks to the use of more granular data, made possible by AI, this year’s model should have worked better than the 2014 one.<p>If anything, it worked worse.</i><p>"If anything"? All the results are available, so it would be easy to put a precise number on this. Measure the Bayesian regret, or just report the winnings if you had used the GS model to bet on the outcomes. Unless it reports some concrete numbers, this article is garbage.<p>It doesn't report any concrete numbers.
Soccer is a sport with a big random component. This is probably why it is so exciting. An average team can beat a better team.<p>The reason is easy to see. The game can be decided by one, two or three key plays. Compare that to basket ball. To win a game you have to consistently score more and defend better. Rarely the game is decided by one or two plays. That only happens when the game is already very tight.
I put money on Belgium (12.0 decimal odds) and Croatia (15.0) after the group stages, where some form was visible, combined with knowledge that they had some of the world's best players.<p>The odds shortened as the tournament progressed, I was able to hedge as the shortened odds made lay betting profitable.<p>(High variance in football outcomes means there's no guarantee of profit, I don't bet big sums.)
Financial modeling is about risk adjust return. Because GS knows they can not determine with certainty the outcome of a given investment, they diversify and hedge. Most of all, GS is a market maker, the equivalent of a bookie. To say that GS’s models “didn’t come close” is to ignore all the ways in which such a grading scheme is different than GS’s actual business model. If their WC prediction efforts acted as anything more than a fun spirited PR project, it was likely that GS wanted to somehow keep its employees engaged and adding business value during the WC which they otherwise would have been certainly watched all month.
In addition to the many other problems with this article, I would like to point out that if, somehow, Goldman Sachs had managed to create a model that could accurately predict the results, the game of soccer would have to be changed to make it more unpredictable somehow. It is intrinsic to the nature of sport that, in order to be entertaining, there has to be a realistic chance for more than one team to win. Not many people (even from the winning country) would bother watching if it were accurately predictable.
Good... There was this discussion thread few days back on HN<p><a href="https://news.ycombinator.com/item?id=17509407" rel="nofollow">https://news.ycombinator.com/item?id=17509407</a><p>Did this investment bank use same set of algorithms that they use for financial predictions?<p>...And then I remember there was this Octopus[1] who used to predict winners with 85% accuracy<p>[1]<a href="https://en.wikipedia.org/wiki/Paul_the_Octopus" rel="nofollow">https://en.wikipedia.org/wiki/Paul_the_Octopus</a>
You'd have to run this world cup thousands of times by simulation, running it a single time and determining the results are not in line with the model is meaningless and silly.<p>It's as silly as saying my claim for the odds of nearly perfectly modelling a coin toss (approximately 50/50%) is wrong because a series of 10 coin tosses show different results from my model. The model is not any less correct.
It's as good time as any to plug in EA's simulation results: <a href="https://www.easports.com/fifa/news/2018/ea-sports-predicts-world-cup-fifa-18" rel="nofollow">https://www.easports.com/fifa/news/2018/ea-sports-predicts-w...</a>
Duh. Looks like there's a fundamental misunderstanding of how statistics works all around. The probability of an event does NOT predict a particular outcome. Ever. It only says that if the experiment is performed again and again and again, like a few thousand times, then X% of those will match that probability.<p>If I toss a fair coin you cannot predict the next outcome. You can only say that if I toss the coin a 1000 times, then close to 500 are going to turn up heads, and another 500 are going to turn up tails.<p>It was stupid of Goldman Sachs or whoever to predict an outcome. It was stupid of anyone else to lend credence to that prediction.<p>Hopefully, Goldman Sachs is not relying on prediction of singular outcomes to make their investment decisions. I don't think they are. Probably just marketing brouhaha to ride the soccer wave. Although I'm not sure if that worked as expected.
They don’t have an edge like they do in their bread and butter markets, combined with a small sample set == high probability of a single year of sports predictions falling over like this
If GS would need to bet money - their actual business model would likely be to sell a bit of each higher probability losers (less risk) vs. buy big on a projected winners (higher risk).
This site is a good counter-example for website optimization: While it uses many assets, so a CDN domain makes sense, it spreads them out thinly. It loads over 100 CSS files, most of which are below 1K. Similarly it loads approximately 30 JS scripts, most of which are just a few K each. This is mitigated to a large extent by using HTTP/2.0, which permits a few dozen or so parallel requests, but it still means that a repeated load of the page takes 2-3 seconds. (Without HTTP/2.0 this probably takes ages, since browsers open only a few connections to each origin at most). There is also almost no difference between reloading with and without the cache.
In case anyone was interested here is a table of how likely the model thought each team was to make it through any particular stage[0] along with the stage that that team went out in and the probability that the model gave for that particular outcome (i.e. [probability of making it through the final stage they made it through] - [probability of making it through the stage they went out in]).<p><pre><code> Groups Round_16 Quarters Semis Finals Out_In Probability
Brazil 87.5% 60.8% 42.0% 27.9% 18.5% Quarters 18.8%
France 81.4% 58.4% 36.6% 19.9% 11.3% Won 11.3%
Germany 80.5% 49.5% 30.5% 18.8% 10.7% Groups 19.5%
Portugal 75.2% 52.8% 32.2% 17.3% 9.4% Round_16 22.4%
Belgium 78.5% 51.1% 27.7% 15.8% 8.2% Semis 11.9%
Spain 72.3% 50.1% 28.8% 15.4% 7.8% Round_16 22.2%
England 73.1% 46.6% 24.4% 13.4% 6.5% Semis 11.0%
Argentina 79.7% 44.2% 24.1% 11.8% 5.7% Round_16 35.5%
Colombia 74.9% 37.3% 17.0% 8.5% 3.7% Round_16 37.6%
Uruguay 74.4% 34.6% 17.2% 7.2% 3.2% Quarters 17.4%
Poland 68.5% 30.5% 12.8% 5.8% 2.3% Groups 31.5%
Denmark 47.8% 26.3% 12.4% 5.2% 2.0% Round_16 21.5%
Mexico 52.0% 23.2% 10.5% 4.9% 1.9% Round_16 28.8%
Sweden 45.9% 19.4% 8.3% 3.7% 1.3% Quarters 11.1%
Iran 35.4% 18.1% 7.2% 2.6% 0.8% Groups 64.6%
Peru 37.3% 17.2% 6.8% 2.5% 0.8% Groups 62.7%
Australia 33.5% 15.4% 6.3% 2.3% 0.7% Groups 66.5%
Russia 47.9% 16.3% 6.0% 2.0% 0.7% Quarters 10.3%
Croatia 49.8% 16.9% 6.3% 2.1% 0.6% Finals 4.2%
Switzerland 52.8% 15.9% 6.1% 2.0% 0.6% Round_16 36.9%
Iceland 45.2% 15.1% 5.6% 1.8% 0.5% Groups 54.8%
Costa_Rica 36.8% 13.3% 4.7% 1.6% 0.5% Groups 63.2%
Serbia 32.9% 12.1% 4.5% 1.5% 0.5% Groups 67.1%
Japan 36.5% 12.8% 3.8% 1.3% 0.4% Round_16 23.7%
Saudi_Arabia 43.4% 12.7% 4.2% 1.3% 0.4% Groups 56.6%
Tunisia 35.2% 13.3% 4.1% 1.3% 0.4% Groups 64.8%
Egypt 34.4% 8.7% 2.5% 0.7% 0.2% Groups 65.6%
South_Korea 21.6% 5.9% 7.1% 0.5% 0.2% Groups 78.4%
Morocco 17.1% 6.8% 1.8% 0.5% 0.1% Groups 82.9%
Nigeria 25.2% 6.5% 1.7% 0.4% 0.0% Groups 74.8%
Senegal 20.1% 4.9% 1.2% 0.3% 0.0% Groups 79.9%
Panama 13.2% 3.3% 0.5% 0.1% 0.0% Groups 86.8%
</code></pre>
[0]: Exhibit 2 in <a href="http://www.goldmansachs.com/our-thinking/pages/world-cup-2018/multimedia/report.pdf" rel="nofollow">http://www.goldmansachs.com/our-thinking/pages/world-cup-201...</a><p>Edit: Fix copy-paste errors and atrocious maths.
While I agree that it's somewhat silly to try and predict a word cup winner like this (and I suspect it was just a bit of fun anyway), there is one other reason that could explain why all these attempts got it so wrong.<p>Cheating.<p>Before people start booing, let's not forget where this tournament is being held, and all the other nefarious things that country has been up to recently.