IMO this 2016 article really hasn't aged well. It turned out that architectural improvements really did matter, and there was still loads of low-hanging fruit. His prediction that bayesian approaches (which were the topic of his PhD) would turn out to be fundamental, has not turned out to be true so far (although they do have their place).<p>(I think in general when people say that their special area of study is particularly important, it should be taken with a grain of salt!)
This is terrible and frankly self serving advice.<p>"Don't build on top of deep learning. Build on top of MCMC-like methods"<p>I used to do research into such methods. That game is over. It's a massive waste of time at the moment. The whole idea is how do we import what was good about those methods into the modern deep learning toolkit. How do I sample from distribution with dl? How do I get uncertainty estimates? How do I compose models, get disentangled representations, get few shot learning, etc.<p>The idea that people should go back to tools like MCMC today is pretty absurd. That entire research program was a failure and never scaled to anything. I say this of my many dozens of papers in the area too.<p>I would never give this advice to my PhD students.<p>Maybe in a decade or two someone will rescue MCMC like methods. In the meantime your PhD students will suffer by being irrelevant and having skills that no one needs.
My general belief, is that the best way to learn at the frontier of something is to pick a problem or a goal and try to solve it. Then you will learn what is in the way of getting that done.<p>Unless you already have deep expertise, I think it's a bad idea to pick a research area and just go and research that. You won't have intuition about why it's a good thing to research. However, you can have intuition about real world problems and the solutions you want to see, and then work backwards to what you need to research.
It is interesting to look back and evaluate the preferences / intuitions prominent researchers had in the field (most of whom started their careers experimenting with MNIST-scale of data, at best)<p>With access to unfathomable amounts of data, especially over the last couple of years, the game changed entirely and is not seeming to cool down anytime soon.<p>The field, certainly, values engineering a lot more than it used to, and it is exciting to see how major advances together with open-source contributions are going to take us
Maybe if you stick to purely theoretical stuff, it's easy. Actually building real systems that work and add value using deep learning isn't easy. There are so many gotchas.
<a href="https://www.inference.vc/we-may-be-surprised-again/" rel="nofollow noreferrer">https://www.inference.vc/we-may-be-surprised-again/</a><p>The author wrote this article one month ago, and he mentioned where he thought wrong about DL
Pretty straightforward case of the curse of knowledge (<a href="https://en.wikipedia.org/wiki/Curse_of_knowledge" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Curse_of_knowledge</a>), in my opinion.