To me, academia has always been about developing proof-of-concepts for techniques that industry adopts years later. These proof-of-concepts are intricate but small, so they require a lot of ingenuity but not much resources or man-hours (a company can also hire far more employees than a professor).<p>AI is no exception. All evidence suggests that current LLMs don't scale (EDIT: they do scale, but at a certain point somewhere around GPT4 the scaling slows down very quickly), so we need new techniques to fix their (many) flaws. A proof-of-concept for a new technique doesn't need hundreds of millions of dollars to demonstrate huge potential: such a demonstration only needs a small model using the new technique, a small model using the current SOTA, and some evidence that it scales (like slightly larger models that show the new model isn't slowing down its scaling vs SOTA).<p>Academia is also about creating better explanations for things we already have. So researchers don't even need to develop new models, they can simply create small existing models and demonstrate some new property or explanation to get a worthwhile result. We probably need better explanations for how and why current LLMs work in order to create the better models mentioned above.<p>EDIT: At least how it's supposed to work, you don't even need to show success either. Academics merely need to show a new technique with the <i>potential</i> to improve the SOTA. Even if the technique is a huge failure, the work still advances the future of AI, by contributing to our understanding of models (what works and what doesn't) as well as removing one potential option.
I am an AI researcher working for a small company. When ChatGPT came out and people started to solve all sort of problems with prompting, I questioned my role and reasoned that in future middle-level researches will lose their jobs and if I want to stay in this business I have to upgrade my skills and possibly start publishing papers and get a PhD. Since in future only big tech companies would do research in AI and many automation problems of small companies would be solved with foundation models. And the competition for research jobs at big companies would be very tight.
There was a related article [0] about this in the WSJ a while back. Here are a couple of relevant quotes from that article:<p>"Despite this progress, we remain concerned that there is a disproportionate amount of interest by policy makers in the voices of industry leaders rather than those in academia and civil society."<p>"Furthermore, to truly understand this technology, including its sometimes unpredictable emergent capabilities and behaviors, public-sector researchers urgently need to replicate and examine the under-the-hood architecture of these models. That’s why government research labs need to take a larger role in AI."<p>[0] <a href="https://www.wsj.com/tech/ai/artificial-intelligence-united-states-future-76c0082e" rel="nofollow">https://www.wsj.com/tech/ai/artificial-intelligence-united-s...</a>
> Obviously there are limits to what a small model is capable of doing<p>This is not obvious to me. I mean yes, limits exist, but I seriously doubt that we are anywhere close to them. This is exactly the kind of question that needs academic research!<p>I feel that our current models waste tons of compute and memory and there ought to be room to optimize them by 10x or even much more with new techniques.
I know a handful of computer science academics personally, and each of them has founded at least one startup. They can go on sabbatical easily, do the startup, then come back once they sell it or it fails. It's a pretty sweet safety net.<p>If I was an AI professor, I would go get a $1 million comp package on sabbatical then come back. The bubble will likely burst in 1 to 3 years, so you need to get paid now.
I think this is incredible advice.<p>I have and continue to take approach IV and VIII[0], and its worked very well for me. Especially now that my research is investigating LLMs[1], scaling down and using architectures that focus on performance is paying dividends.<p>Working in a niche applications (molecular biology) is also a great way to remain sane (although it comes with its own problems).<p>[0] "Scale down" and "Small models"<p>[1] Well, "large" is a relative term here.
As someone in an adjacent field who tried to keep up with the state of the art until the late 2010s, I really relate to this article. I'm reminded of another interesting personal story about AI-induced academic depression <a href="https://arxiv.org/abs/2003.01870" rel="nofollow">https://arxiv.org/abs/2003.01870</a>. I think it's valuable to record things like these.
The premise is bizarre. Academia is a sanctuary for people who can't get paid to do research because it's not economically exploitable, but is considered socially valuable. Academics are deeply lucky to have this option. Most of us can't get subsidies for our passion work.
There's no reason to be "depressed" about corporations throwing mountains of money at you to come out of the sanctuary.