Choose Your Weapon: Survival Strategies for Depressed AI Academics

As a theorist in AI, the premise is funny but also really sad. And yet it is a very pervasive premise in the field.<p>OpenAI is not doing science. They are building a big shiny thing, showing it off, keeping it closed, and making money off of it. That is not part of the scientific process.<p>It is as if, every time SpaceX launched a rocket a little bit higher, every aerospace department ooh&#x27;d and aah&#x27;d and threw up their hands: &quot;we don&#x27;t have the resources to build a bigger rocket; how can we do science?&quot;<p>Unfortunately, the research field of AI is diseased: it places way too much value on showing off big shiny things above progress in scientific understanding. But there is a ton of progress available to be made by small teams on small budgets. There is so much we don&#x27;t understand about neural networks alone, even small ones.

Academics can’t put themselves on a pedestal for being revolutionaries who care about the ultimate truth the most and at the same time whine when an advance comes along.<p>These CS academics need to understand that this is what it feels like to be in other fields like physics or bio where you can’t do jack unless you have costly equipment. This is what people in developing countries deal with all the time.<p>And people will build the next GPT or whatever and even that’ll get boring. The people in big tech have to pivot and do whatever the economy demands. Whereas the academics can go back to writing papers like nothing changed.

Obviously AI academics are uniquely challenged by the &#x27;scaling is (nearly) everything&#x27; reality, but I feel like a lot of the mixed emotions being expressed towards the latest results are also because we&#x27;re actually seeing the mystery of self starting to unravel. Like any good mystery, the fun was in the build up and as we move towards a resolution there&#x27;s a bitter sweet aspect to the slightly mundane reality of it &#x27;merely&#x27; being an emergent property of large networks. Of course, any rational person might have expected this given how the one working example came into existence. In general it all looks like a pretty resounding endorsement of the David Chalmers position to me.

&gt; If we have learned one thing from deep learning, it is that scaling works. From the ImageNet [15] competitions and their various winners to ChatGPT, Gato [13], and most recently to GPT-4 [1], we have seen that more data and more compute yield quantitatively and often even qualitatively better results. (By the time you are reading this, that list of very recent AI milestones might very well be outdated.). Of course there are improvements to learning algorithms and network architectures as well, but these improvements are only really useful in the context of the massive scale of experiments. (Sutton talks about the “Bitter Pill”, referring to that simple methods that scale well always win the day when more compute becomes available [18].) A scale that is not achievable by academic researchers nowadays. As far as we can tell, the gap between the amount of compute available to ordinary researchers and the amount available to stay competitive is growing every year.<p>&gt;This goes a long way to explain the resentment that many AI researchers in academia feel towards these companies. Healthy competition from your peers is one thing, but competition from someone that has so much resources that they can easily do things you could never, no matter how good your ideas are, is another thing.<p>This sounds like teenagers whining they can&#x27;t all be popular. Science isn&#x27;t a competition to be won so that you can get praise and attention. Science exists to discover things that then hopefully are useful to people. If someone that isn&#x27;t you discovers something useful that isn&#x27;t bad because then you can&#x27;t discover it. It is good because that is a problem solved.<p>The resources requirements FOR SOME SPECIFIC PROBLEMS have gone up to a ridiculous degree, but there are plenty of problems left to solve. In fact a new one that is at least as important has been created: Replicate current results with less hardware&#x2F;parameters&#x2F;whatever.<p>The difference between having GPT4 as a slow and costly service that requires network calls vs having it locally with almost no cost will be a huge achievement. Stop sulking and get to work!

I can imagine AI academics are in a tough spot. However what about the existential angst the rest of us - who don&#x27;t even do anything AI related on a day to day basis - are feeling? I think big changes are coming and it&#x27;s not gonna be pretty.

&gt; the grant funding structure is such that it rewards safe and incremental research on popular topics [...] Therefore, universities should probably avoid making grant funding a condition for hires and promotions<p>There you go. Just fix this idiotic &quot;publish (NeurIPS) or perish&quot; attitude already!

I think it&#x27;s always been the case that if industry <i>really</i> gets interested in making progress in some particular area, they make it, and way faster than academia ever could. Not just in computer science but also in fields like chemistry or engineering. Academia is great for doing basic science where there is no profit motive for industry to do something about it. That&#x27;s what the niche&#x2F;&quot;few care about&quot; sections talk about, where IMO the strength of academia lies. computer science is I think in general comparatively in a good position compared to other sciences in how easy it is to get funded.

This papar is really interesting, especially so when you scroll to the Reference part ;)

After a PhD@MIT and then entering industry, it has been my general understanding that industry is ahead on any scientific fronts that are profitable in fields that have a progress&lt;&gt;profit feedback loop.<p>Academia still plays an important training ground role, and it could shift focus to those areas that could be profitable but require coordination that is generally not feasible among certain institutions.

Better give up now as a scientist so you got an early lead into becoming a chef. Or one of the other jobs where a human touch is still valued.

That&#x27;s the first time I&#x27;ve been rick roll&#x27;d reading an academic paper :) (citation 1 for GPT-4)

I expected this to include more resources like TPU Research Cloud, but it doesn&#x27;t mention even TRC itself. Disappointing.<p><a href="https:&#x2F;&#x2F;sites.research.google&#x2F;trc&#x2F;about&#x2F;" rel="nofollow">https:&#x2F;&#x2F;sites.research.google&#x2F;trc&#x2F;about&#x2F;</a>

This paper is<p>1. Really a blog post disguised as a paper.<p>2. Silly.<p>3. Excellent.

I love that the citation to GPT-4 is a Rickroll :)

I think this paper is a joke but like all good jokes has a grain of truth. I would imagine that most AI academics are very excited by recent developments, if you worked on conversational language models you are more likely to view recent progress at OpenAI as a vindication than a threat. That said there must be some sadness that it wasn&#x27;t you.

Don&#x27;t worry. Soon governments will start pouring money into this field. They still won&#x27;t be competitive, but at least there will be enough computing power to play with.

I like how they release what reads more like a blog post as a paper.

AI profs should start working with the bio folks now and making thinking materials that are orders of magnitude more power efficient than silicon.

Thus is medium shit post at best.. why is it on arxiv

The only limitation of this scientific article is the fact that it is not two-columns format.

Gotta love a paper which cites Rick Astley&#x27;s &quot;Never Gonna Give you Up&quot;.<p>Properly, for that matter.

AI fatigue is the new JavaScript fatigue.