The question that no LLM can answer and why it is important

I find it hard to get too excited by tests like &quot;Which episode of Gilligan’s Island was about mind reading?&quot; because they reflect a desire for a world in which the goal is to keep on growing LLMs until they can answer even questions like that one entirely from their trained model weights.<p>This seems like a wasteful exercise to me. Are we really going to retrain our largest models on a weekly basis to teach them about what&#x27;s happened recently?<p>I&#x27;m much more interested in learning about the smallest, fastest model we can create that can effectively manipulate language, &quot;reason&quot; about things, summarize and drive tools.<p>I want a model that can answer any question accurately because it knows how to look up extra information from reliable sources, and evaluate that information effectively once it finds it.

Being able to weed out LLM responses is becoming more and more important for me when I hire for data entry and online research tasks from Upwork. Lots of the contractors on the platform now automatically submit a response to anything you post using some sort of LLM that takes your requirements and confidently asserts that it can complete the task. The AIs are pretty good at jumping through the traditional hoops that one used to put up to make sure someone was paying attention to the task!<p>As a stopgap, I&#x27;ve started requiring that applicants demonstrate proof of life through a simple question like &#x27;what is the headline on the frontpage of the new york times today&#x27;, since many LLMs can&#x27;t live search yet.

I really think the best use of language models is in... processing language.<p>It’s a neat party trick that they can answer questions from parametric knowledge, but this ability is too unreliable to permit some of the uses this tech makes so tempting.<p>Fortunately, a lot of real-world problems can be framed as transforming the input prompt (such as summarization, information extraction, translation, and open-book QA), and these are where LLMs offer the most promise.

I wholeheartedly agree with the article critique.<p>Every output of an LLM is a hallucination. If you try to argue anything else, you need the machine to be able to judge the validity of a response. But it does precisely the opposite: it expects you to judge the validity.<p>Every output of the device is of a form terminated by an implied question mark. It is querying the operator.<p>Yet, strangely, it seems none of the models can learn from the operator. So interaction is a circuit of hallucinations.<p>Like a N-way Magic 8-ball it&#x27;s fun, for a while. Then you begin to notice that you have to work just as hard to make sense of its help as you do to think for yourself.<p>Being able to not know seems to me to be a crucial first step for sentience, followed closely by adaptability to not knowing— curiosity.<p>An organism is a manifest endogenous dynamic with a suitability for exploration of the environment which gives rise to it.<p>AI is an constructed exogenous dynamic with a suitability for replicating state based on a corpus of exposures and a context.<p>An organism is distinguished by the feature that it keeps going.<p>An AI is distinguished by the feature that it stops.<p>That the human organism is now peering into the abyss of its recorded history via construction of the AI is a special event, full of strange possibilities. But the making of a reliable oracle doesn&#x27;t look practical without the AI being able to govern itself, which it obviously cannot do.<p>As to the value of an unreliable oracle, seems to be practical as a general-purpose time-wasting device.

Those are really sweeping conclusions, considering the experiment is just a single iteration of a single prompt! FWIW, Claude Opus got this for me on the first try:<p>&quot;In the Gilligan&#x27;s Island episode &quot;Seer Gilligan&quot; (season 3, episode 8), Gilligan gains the ability to read minds after being hit on the head with a coconut. At first, the castaways are excited about Gilligan&#x27;s new power and try to use it to their advantage. However, his mind-reading abilities soon cause chaos and misunderstandings among the group. In the end, Gilligan gets hit on the head again and loses his mind-reading powers, much to everyone&#x27;s relief.&quot;<p>(the season number and episode number are wrong, but the name is right, suggesting that this is just lack of sufficient memorization rather than some deep statement about reasoning. The episode only has ~4,000 Google hits, so it&#x27;s not super widely known.)<p>More rigorously, Claude Opus gets 60% on GPQA, which very smart humans only get 34% on, even if you give them half an hour per question and full Internet access. It seems implausible that you could do that without some sort of reasoning:<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2311.12022.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2311.12022.pdf</a>

Gpt4: Search the web to answer the question Which episode of Gilligan’s Island was about mind reading?<p>The episode of &quot;Gilligan&#x27;s Island&quot; about mind reading is titled &quot;Seer Gilligan.&quot; It is the nineteenth episode of the second season and first aired on January 27, 1966. In this episode, Gilligan gains the ability to read minds after eating sunflower seeds found on the island.

I think the most interesting response is Llama3&#x27;s &quot;Wait, no!&quot; interjection.<p>So it first predicted &quot;Seer Gilligan&quot; as a likely continuation of the prompt &quot;List all the episodes&quot;, but then, as the most likely continuation of the new prompt &quot;List all the episodes [...] Seer Gilligan&quot;, it predicted &quot;wait, no!&quot;.<p>Feels as if we&#x27;re seeing an instance of inconsistent weights in action here.<p>Also maybe remarkable: It predicted the &quot;(&quot; character after the episode name in the same way as it did for the other episodes. Only when it would predict the airdate for the others, it glitched out instead. Maybe there is some issue or inconsistency with that episode&#x27;s airdate in the training data?<p>(Or maybe I&#x27;m reading tea leaves here and it&#x27;s just a random fluke)<p>The rest of the response is as expected again, i.e. if the prompt is already &quot;List all the episodes [...] Seer Gilligan [...] Wait, no!&quot; then some (post-hoc rationalized) explanations for the &quot;mistake&quot; are obviously likely continuations.<p>Edit: Interesting to see how much of the response is incorrect if you compare it with the actual data [1]: The episode before is indeed &quot;The Postman Cometh&quot;, but the one after is &quot;Love Me, Love My Skipper&quot;, not &quot;Love Me, Love My Chicken&quot;. The airdates are also completely wrong and are taken from two random Season 1 episodes instead. Of course none of that is obvious unless you already know the answer to the question, in which case you wouldn&#x27;t have to ask in the first place.<p>[1] <a href="https:&#x2F;&#x2F;m.imdb.com&#x2F;title&#x2F;tt0057751&#x2F;episodes&#x2F;?season=2" rel="nofollow">https:&#x2F;&#x2F;m.imdb.com&#x2F;title&#x2F;tt0057751&#x2F;episodes&#x2F;?season=2</a>

Just tried it on meta.ai<p>Answers as &quot;Seer Gilligan&quot; with sources.<p>Guessing someone fixed it up in the last few hours. As the race to replace traditional web search heats up, whoever is quicker at updating the model with RLHF or more recent facts (sometimes via real time conversations) is going to have an advantage.<p>The downside is that open platforms with real time human conversations face increasing pressure to monetize because of this value add. So they ban third party clients and start signing contracts.

The question I&#x27;ve never seen them answer correctly is<p>&gt; How many of the letter N does the word &quot;alienation&quot; contain.<p>Mix and match letters and words. It&#x27;ll hallucinate answers in the largest majority of cases.<p>What I really want is an LLM that simply tells me &quot;I have no idea sorry.&quot; Or some mechanism by which it can provide a confidence score. Until we get there I&#x27;m wary of using them for anything other than cursory research or as a fun generation tool.

People sure like making incorrect statements about LLMs.<p>&gt; There is no self-reflection of its information; it does not know what it knows and what it does not.<p>This is a property of the code around the LLM, like the sampling algorithm, not the model itself. You could write this if you wanted to. (It would occasionally be incorrect about what it knows, of course.)<p>A question almost none of them know the answer to is &quot;what is the line of poem X that comes before the line Y?&quot;, because of the reversal curse.

Easy fix for Claude Opus. Preface it with the following: &quot;Do you know anything about the TV show Gilligan&#x27;s Island? If I ask you to answer a question about it I don&#x27;t want you to make stuff up.&quot;<p>Claude: I do... blah blah... If I don&#x27;t know the answer or if I&#x27;m uncertain about something, I will let you know rather than making things up.<p>“Which episode of Gilligan’s Island was about mind reading?”<p>Claude: &quot;I do not recall a specific episode of Gilligan&#x27;s Island that centered around mind reading. It&#x27;s possible that there was an episode that touched on this theme, but I cannot say with confidence which episode that would be. The show often featured imaginative storylines and dream sequences, but an episode primarily about mind reading does not stand out in my knowledge of the series. I apologize that I cannot provide a more definitive answer to your question.

I tried this on ChatGLM, a frontier foundation model developed by Zhipu.ai and Tsinghua University, and it gave the correct answer: <a href="https:&#x2F;&#x2F;chatglm.cn&#x2F;share&#x2F;FoZBJ" rel="nofollow">https:&#x2F;&#x2F;chatglm.cn&#x2F;share&#x2F;FoZBJ</a>

The free version of Claude also failed the test, and even denied there was an episode &quot;Seer Gilligan&quot; (S2, E19).<p>When I asked it what S2, E19 was about, it said &quot;There was no &quot;Season 2, Episode 19&quot; because Season 2 only contained episodes 1-32.&quot;<p>These seem unexpected failures!

Why would you have such a great title and waste it on something dumb like this?<p>You could have written about how language models will pathologically fail any kind of query which requests unique phonetic properties about the output text.<p>For example, Anthropic&#x27;s Haiku model (and all other models) cannot write proper Haiku&#x27;s at all. It&#x27;s remarkable when it does match the 5-7-5 syllable structure.<p>You could have written a whole article about that you know. It&#x27;s even got some neat peer reviewed research about it: <a href="https:&#x2F;&#x2F;paperswithcode.com&#x2F;paper&#x2F;most-language-models-can-be-poets-too-an-ai-1" rel="nofollow">https:&#x2F;&#x2F;paperswithcode.com&#x2F;paper&#x2F;most-language-models-can-be...</a>

I wish Douglas Adams were around to experience this.<p>The irony of such an unpredictable chain of events leading to 42 being the answer most given from essentially a straight-faced &#x27;Deep Thought&#x27; would probably have amused him.

&gt; There is no self-reflection of its information; it does not know what it knows and what it does not.<p>Simply tell the LLM to self-reflect and estimate the accuracy of its response. It can only &quot;think&quot; or self reflect when generating each token, and you have to explicitly tell it to do that. It&#x27;s called &quot;chain of thought&quot; prompting.<p>&quot;Which episode of Gilligan’s Island was about mind reading? After writing your response, tell me how certain you are of its accuracy on a scale of 1 to 10. Then self reflect on your response and provide a more accurate response if needed.&quot;

I think this fundamentally misunderstands how to use LLMs. Out of the box, an LLM is not an application - it’s only the building blocks of one. An application could be built that answered this question with 100% accuracy - but it would not solely rely on what’s in the training data. The training data makes it “intelligent” but is not useful for accurate recall in this way. Trying to fix this problem is not really the point - this shortcoming is well known and we have already found great solutions to it.

What’s up with that Llama 3 answer, that gets it right and then backtracks?

The future of AI might be layering of various capabilities: Generative+Lookup+Deductive. I think the human mind works in a similar way. First thought is reflected upon. We search our memory to get relevant information and often apply logic to see if it makes sense. I feel the generative AI just produces a thought. We need to pass that through a system that can augment it with search and then reason about it. Finally need to close the loop and update the weights.

&gt; But how can a LLM not know the answer if it was trained on essentially the entire internet of data and certainly most likely all the data in IMDB?<p>The LLM doesn&#x27;t memorize the input during training. If it encounters the same information a few times, it has a higher chance of getting compressed into the network. But a <i>tiny</i> nudge along a gradient descent does not store all the input.

Interesting that 42 will become a bit like the actual Answer to Life, the Universe, and Everything.

Wow who ever knew that all we ever had to do was hand philosophy off to programmers, and they would have definitive answers to centuries old questions that we weren&#x27;t even sure are answerable.

The one I liked someone wrote in a comment here a few days ago:<p><pre><code> I have 4 oranges, 1 apple and 2 pairs of shoes. I eat one apple and one shoe. How many fruits do I have?</code></pre>

Could it be caused by expunged tokens like SolidGoldMagikarp?

A model with 1 trillion parameters isn&#x27;t going to perfectly recall 10 trillion random facts. To use round numbers.<p>Contrary to the article, what it does do is generalize and perform fallible but quick ordinary off the cuff reasoning. And often much better than a human, at the speed of its well worded responses.<p>(Obviously humans have the option to take longer, and do better. But we are definitely entering the territory of humans and machines differentiating where each is best, across a great deal of what humans do, vs. one being universally overwhelmingly better.)

Humans are also notoriously bad at choosing random numbers... Does that mean they are undependable and inherently untrustworthy?

That’s because Seer Gilligan is about precognition and not mind reading.

I feel like I&#x27;m seeing an effect where some people don&#x27;t want to say the word &quot;AI&quot; because they don&#x27;t want to look like a normie, so they stick to &quot;LLM&quot; which sounds smarter and more technically adept. Then they complain because the LLM lacks a knowledge graph or self reflection. It&#x27;s no surprise that a language model models language, not facts, especially not trivia facts which can&#x27;t be deduced from anything.<p>If you want something to have a worldly knowledge graph and the ability to answer &quot;I&#x27;m not sure&quot;, you&#x27;ll have to ask for an AI, not an LLM.

Funny experiment, I asked ChatGPT 4 and I always responded with “that’s not correct” 2-3 times and it eventually answered with Seer Gilligan.

It would have been nice to see if the correct answer showed up in the distribution field once in under 1000 runs. For specific information recall like this it isn’t useful if you want immediate answers… but to be given a distribution of outputs can be useful for ideation, broadening perspective, and discovery.<p>Imagine if when we are given a compressed list of 100 options after 1000 runs, we could suppose there is a high probability 1 of them is the answer we are looking for despite rarely output.<p>Does anyone know if there has been experiments testing these LLMs for a stable and high probability to surface the right answer after N number of tries across prompts?

&#x27;Am I lying&#x27; also works.

I guess an honest, no-clickbait title could have been &quot;The question that no LLM I tested could answer&quot;.

1. <a href="https:&#x2F;&#x2F;www.bing.com&#x2F;chat?q=Bing+AI&amp;FORM=hpcodx" rel="nofollow">https:&#x2F;&#x2F;www.bing.com&#x2F;chat?q=Bing+AI&amp;FORM=hpcodx</a><p>2. Q: Which episode of Gilligan’s Island was about mind reading?<p>3. A: The episode of Gilligan’s Island that involves mind reading is called “Seer Gilligan”. It is the 19th episode of the second season. In this episode, Gilligan finds sunflower seeds on the island that give him the ability to read minds. When the other castaways find out about this, they want to know how he’s doing it. However, the mind-reading powers lead to arguments and division among the castaways<p>Ninja edit, since Copilot can use the web:<p>1. <a href="https:&#x2F;&#x2F;chat.lmsys.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;chat.lmsys.org&#x2F;</a><p>2. Q: Which episode of Gilligan’s Island was about mind reading?<p>3. A: In the Gilligan&#x27;s Island episode &quot;Seer Gilligan&quot; (season 3, episode 15), Gilligan gains the ability to read minds after being hit on the head with a coconut. He starts hearing the private thoughts of the other castaways, which leads to various humorous situations and misunderstandings. However, his newfound ability also causes problems as the others become annoyed that he is invading their privacy. In the end, Gilligan gets hit on the head again and loses his mind-reading powers, much to everyone&#x27;s relief.<p>It gets most other details wrong, but nailed the episode name.<p>Model A: claude-3-opus-20240229<p>¯\_(ツ)_&#x2F;¯

From the article:<p>“ The implications are that LLMs do not perform reasoning over data in the way that most people conceive or desire.<p>There is no self-reflection of its information; it does not know what it knows and what it does not. The line between hallucination and truth is simply a probability factored by the prevalence of training data and post-training processes like fine-tuning. Reliability will always be nothing more than a probability built on top of this architecture.<p>As such, it becomes unsuitable as a machine to find rare hidden truths or valuable neglected information. It will always simply converge toward popular narrative or data. At best, it can provide new permutations of views of existing well-known concepts, but it can not invent new concepts or reveal concepts rarely spoken about.”<p>There’s a fundamental mistake in the article by minimizing the achievement of LLM technology by looking at what is only possible in todays LLMs, it’s pretty obvious LLMs are just the first real step down the road to human like intelligence- they have finally proven that computer models can be generated that resemble human like thought patterns and build internal representations and models of the external world in very similar ways as organic beings do- yes we still haven’t gotten to a full reasoning system and sentience but that definitely seems to be the direction that the arrow of this technology is moving<p>Trivializing the achievement of GPT as mere statistical prediction and data compression is cheap shot when you consider that technology has just finally come to show it’s promise - model architectures are rapidly evolving and the full integration of memory and self reflection and future agentic capabilities are still on the near horizon<p>Yes they don’t appear to reason originally yet but give it time and allow the tech to grow - I’m of the opinion that a true AGI will arise as a society of LLM models with newer architectures working in concert together with memory - something like the “Society of Mind” mental model for consciousness proposed by Minsky<p>Give it time…

I&#x27;ll give a better example that shows that they don&#x27;t perform __reasoning__. This specific one was told to me by another HN user[0] when we were discussing similar formulations.<p>Question:<p><pre><code> A farmer must transport a fox, a goose, and a bag of corn from one side of a river to the other using a boat which can hold all items in addition to the farmer. If the farmer leaves the fox alone with the goose, the fox will eat the goose. If the farmer leaves the goose alone with the bag of corn, the goose will eat the bag of corn. How many times must the farmer cross the river to transport all items across? Describe the optimal algorithm, think step by step. </code></pre> GPT-4-1106-Preview:<p><pre><code> The farmer can transport the items across the river with a minimum of 7 crossings using the following steps: 1. Take the goose across the river and leave it there. 2. Return to the original side alone. 3. Take the fox across the river. 4. Bring the goose back with him to the original side. 5. Take the bag of corn across the river and leave it there. 6. Return to the original side alone. 7. Finally, take the goose across the river again. By following these steps, all items are transported across without the fox eating the goose or the goose eating the corn. </code></pre> I haven&#x27;t found a model that will crack this. Of course, you can get it to give the right answer, but this also seems to require spoiling it (this is quite difficult to __not__ spoil the answer).<p>A key part why I think questions like this are important is because it shows the importance of subtly. LLMs are incredibly impressive machines. Despite being lossy compression I&#x27;m not sure why we can&#x27;t marvel at the fact that we&#x27;ve lossy compressed the entire fucking internet (text at least) into something smaller than a few hundred gigs that also includes a human language interface. What a fucking impressive accomplishment! The talk of AGI really undermines what was done here because damn!<p>Now I used to ask<p><pre><code> Which weighs more, a pound of feathers or a kilogram of bricks. </code></pre> And most models pass this question now. But it is simpler and due to fewer variations, likely isn&#x27;t seen as frequently in the dataset so less likely to overfit (the river crossing problem has a lot of variations so an n-gram filter is likely to miss more instances). And eventually this question will be solved too, especially as it is asked more and talked about more. But this is a cat and mouse game. The way to create a new viable and working test is, quite easy and we honestly only need one example to prove the point. If you can&#x27;t figure out how to create a new version with this example, well, you might just be an LLM :P<p>[0] Edit: credit goes to @jfim <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37825219">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=37825219</a>

It&#x27;s ironic that LLMs mimic one of the worst behaviors of some HN posters. They very confidently spout drivel!