You need a mental model of LLMs to build or use a LLM-based product

&gt; LLMs are booksmart Harvard graduates who can Google anything<p>I wouldn&#x27;t say this. Internet savvy Harvard graduates have common sense. They can look at something and mostly infer &quot;wait, that&#x27;s not right,&quot; and error correct, or admit low confidence in the answer.<p>I like to think of LLMs as hardcore improv actors. They have a script (the context). Their burning desire is to continue that script. And they will just roll with the best continuation they can, <i>whatever</i> that continuation is. OpenAI&#x27;s augmentations hand them a very dynamic script, but at the end of the day they could just be madly improvising for all you know.

What a plot twist that the appropriate mental model for LLMs is one with a near-fatal flaw that&#x27;s neatly solved by SourceGraph&#x27;s product.<p>Edit: Not trying to be dismissive, this is actually giving you a bad mental model because it understates the capabilities of LLMs for the sake of selling their product. The way they use the LLM in their examples is completely uninteresting and adds no actual information beyond what was just fed to it. Repeating back the question would give the same usefulness! Surely they can come up with a better sales pitch for why you need &quot;Cody&quot; than that you can&#x27;t tell what parameter names in a function call mean.

LLMs are not booksmart Harvard grads or anything else. They are extremely complex statistical models doing next-token prediction. That&#x27;s it - that&#x27;s all. If you want a proper mental model of LLMs, you need to understand this - the thing you&#x27;re doing is text prediction. You do not have a partner engaged in cognition, you have a ludicrously complicated language model trained on a large corpus of data. If the bulk of that data happens to assert that the sky is blue, the model is likely, but not guaranteed, to finish the sentence &quot;What color is the sky&quot; with &quot;Blue&quot;. That&#x27;s it. That&#x27;s the trick.

&quot;Think of LLMs as booksmart Harvard graduates who can Google anything to answer any question you ask them.&quot;<p>My version of this is similar: think of them as an extremely self-confident undergraduate with an amazing memory who has read all of Wikipedia, but is occasionally inclined towards wild conspiracy theories. And is equally convincing talking about both.

My mental model for LLMs was built by carefully studying the distribution of its output vocabulary at every time step.<p>There are tools that allow you to right click and see all possible continuations for an LLM like you would in a code IDE[1]. Seeing what this vocabulary is[2] and how trivial modifications to the prompt can impact probabilities will do a lot for improving the mental model of how LLM operate.<p>Shameless self plug, but software which can do what I am describing is here, and it&#x27;s worth noting that it ended up as peer reviewed research.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;Hellisotherpeople&#x2F;Constrained-Text-Generation-Studio&#x2F;tree&#x2F;main">https:&#x2F;&#x2F;github.com&#x2F;Hellisotherpeople&#x2F;Constrained-Text-Genera...</a> [2] <a href="https:&#x2F;&#x2F;github.com&#x2F;Hellisotherpeople&#x2F;Constrained-Text-Generation-Studio&#x2F;blob&#x2F;main&#x2F;pictures&#x2F;list_of_vocab_that_fulfills_constraints.JPG">https:&#x2F;&#x2F;github.com&#x2F;Hellisotherpeople&#x2F;Constrained-Text-Genera...</a>

I have a new term: Mono-objective societal collapse - MOSC. MOSC is the scenario where everyone is trying to achieve &quot;AI&quot;, and some or all attempts fail, and the result is base-infrastructure gaps which can affect a small group of individuals or be an existential threat to society.<p>You might say, why is a new term needed? My answer: this article is &#x2F; has been done and redone ... is it written by an LLM and everyone is just recycling it? Or are actual humans spending their time this way?

This article comes across more than a little condescending to me. It vaguely alludes to some of the big criticisms of LLMs (e.g. hallucinations) but then focuses on the specific of &quot;the LLM will give a useless answer if it doesn&#x27;t have access to the necessary information&quot;. I mean... duh. Is anyone complaining about that, especially programmers?

This is content marketing for their AI code assistant Cody, and doesn&#x27;t offer much new information otherwise.

I don&#x27;t think users understand how LLMs are different are different from search engines, especially for information retrieval. Someone I know well has been using ChatGPT and Bard for months, and was surprised that they don&#x27;t just use the Bing&#x2F;Google search engine indexes behind the scenes. The idea that LLMs are a bunch of frozen matrices is not obvious.<p>It&#x27;s hard to communicate that it&#x27;s better to rely on LLMs for some classes of &quot;reasoning&quot; or language tasks vs simple information retrieval, particularly given retrieval does work well much of the time.

If one needs AI to tell them how to read a method signature, they are in the wrong job line.