Long story short:<p>- The LSVRC "visual recognition" competition has a rule in place, that limits to twice per week how often each contestant can run their entries to the contest against the ImageNet dataset.<p>- The Baidu team run their tests much more often, claiming that they understood the limit was placed on a person basis and not per team.<p>- This rule is in place because more frequent submissions somehow distort the quality of the test, by switching its emphasis to overlearning (adapting too much to the specific data included in the test dataset) instead of true algorithmic advances, which can influence the whole machine learning discipline given this contest's high profile. ( * )<p>- As a result, the whole Baidu company has been banned from the competition for a year.<p>( * ) ("The Baidu team’s oversubmissions tilted the balance of forward progress on the LSVRC from algorithmic advances to hyperparameter optimization.")
To translate to a more familiar domain, think about the SATs. After a period of study, students take the SAT where they get back the composite score of how they did, but never the actual answers to the questions on the test.<p>Now imagine a student can take the test repeatedly over the space of a few days, and can use the score to reverse engineer the answers to the questions. They can put in random answers and note which ones cause the score to go up. Of course the real life SATs don't allow this, and they change up the questions to prevent this sort of cheating. If this were possible, our enterprising/cheating student can derive the complete answer key over time, noting the changes in scores for each run. And once they have the key, they can ace the test. No longer is it a test of their aptitude, but rather of their knowledge of the answer key.<p>This scandal is analogous to this albeit contrived example. With an ML testset, it's not possible to change the data because you want it standardized so you can evaluate improvements that new approaches may bring. It's the only way to have a meaningful yardstick to measure against. Thus, the only way to prevent such gaming is restricting multiple submissions, so that you can't do 'hyperparameter optimization' - i.e. overlearn on the testset.<p>That's why it's cheating - it's not a measure of how well your algorithm did, but rather on how well you reverse-engineered the answer key. It's a huge disservice to the field and the people who did this should be ashamed of themselves.
Idle speculation:<p>> "The key sentence here is, 'Please note that you cannot make more than 2 submissions per week.' It is our understanding that this is to say that one individual can at most upload twice per week. Otherwise, if the limit was set for a team, the proper text should be 'your team' instead," Wu wrote.<p>I wonder whether, to a native Chinese speaker, this really does sound like it's talking about individual people, and saying "you" when one means "your team" seems really bizarre. Can any Chinese speakers weigh in?<p>(Even stipulating this, the affair still sounds more like malice than incompetence on the part of Wu.)
Congratulations to the author for both of his last 2 posts making it to the HN front page!<p>This explains the need to drill home the train-test idea from the last post. I hadn't thought about this before but multiple submissions do amount to multiple peeks at your held-out test set, which is a huge ML no-no.<p>I don't know much about LSVRC, but doesn't the way Kaggle work prevent this? AFAIR you get a "public" test-score which is used for the leaderboards, but once the deadline for submissions is up, each submission is evaluated on a held-out test set giving you a "private" score. Now that I think about it, I'm not sure how that works, I guess the accuracy they show you as your public score is only on part of the submitted rows? Regardless of how that's done, could the LSVRC organisers not do something similar?
> there are almost no papers focusing on 3 or 4-layer CNN’s these days, for example<p>What's the 'best practices' for the number of hidden layers in a CNN? 1 or 2 hidden layers?
So how would a better dataset look like? Does bigger always equal harder? What are the criteria to measure that?<p>And wouldn't video datasets be somewhat easier to analyse given the fact that you have multiple frames of the same object?
Nice article on why it is cheating (from a mathematical sense, independent of language) to get scores from the hold-out leader board too many times (plus some methods to mitigate the effect): <a href="http://arxiv.org/abs/1502.04585" rel="nofollow">http://arxiv.org/abs/1502.04585</a> .
As long as you make the public leaderboard set small, and the private one shot leaderboard set very large, the number of submissions matters very little in the final rankings. The only real issue is hand labeling the public leaderboard set to augment training data.
Meh.. I'm not so sure about this.<p>Didn't people claim "cheating" back when the first compilers started doing data flow analysis too?