Kaggle Post-Mortem: The dangers of overfitting

Kaggle is doing ML wrong. It has the opportunity to teach some people about these perils, this blog post being the beginning of such a realization, but I feel terrible for anyone with a hard problem who invests in a funded Kaggle competition.<p>Black box machine learning is unlikely to improve a meaningful metric beyond a stock SVM or random forest. It's possible to tune an algorithm to achieve arbitrarily good training set performance, of which you can include the small "test set" evaluation, but this does not make a generalizable, practical tool.<p>Opportunities might exist if you have the opportunity to feature engineer with domain expertise in order to keep improving your Bayes error. Kaggle is designed not to allow for this, though.

And the irony of it is that the author is warning us against the dangers of overfitting while overfitting the same data that proves his point.<p>I'm referring to the blue lines in the scatterplots of the rank vs. no of submissions: those unidentified curves should have been straight lines. What's the point of having some high degree polynomial there?

Reading "Thinking, fast and slow" right now, it resonates strangely with this article.<p>For those not in the knowing, this book proves how most of our reasoning is false most of the time, shows that people vote based on facial features, that success is luck, that the world is out of our control, that experts are worse forecasters than monkeys randomly picking options, and the better the experts the worse the forecast. Depressing.<p>Edit: one thing reassuring is that it explain rationally how random people moderately clever and sometime plain despicable can become the head of very successful companies.

Not surprising at all - kaggle (as currently implemented) is a fundamentally broken model. On top of the rather unpleasant "everybody pays auction" or "winner take all" system, they have a severe problem with metrics - the majority of the datasets are not anywhere near large enough to give stable out of sample error estimates, which means that in many cases the "winners" are barely better than random.<p>Perhaps they might be onto something with "kaggle prospect", but unless they pivot in some creative new direction, it's hard to see the service being very useful.

I've been competing in the Facebook competition and submitting twice a day. I've managed to drag up into the 90th percentile of scores but I wonder how much the final leaderboard will resemble the daily leaderboard on Tuesday when the competition closes. I'm really looking forward to a post-mortem of this particular problem, since the data was basically featureless and the most successful approaches seem so similar.

From a business perspective, it doesn't matter if Kaggle is flawed. What matters is whether they can convince customers to buy in to their ideas and fund their business.<p>In other words, more Snake Oil from the computer industry. In some (most?) cases, the salesmen may even believe that the Oil works. That is, they may not be acting fraudulently, but just foolishly (as are their clients who believe the hype).<p>Overconfidence in math and computers to solve problems that are not suited to be solved this way. In some ways this is partly responsible for the global financial mess. Reliance on models that quants and their bosses would defend vigorously, because they work to make money for the bosses (because clients believe the hype), but which are, overall, in the long run, not sound.

Why even look at these scoreboards?