> If you can't see the value of LLMs at all at this point, you're holding it wrong.<p>> Sure, LLMs hallucinate. Said cynically, they "lie." You need to add lots of context, or enable Web Search to ground the answers in external sources, or connect MCP servers, or or or... You can't just ask ChatGPT to "make me a fitness program," you need to enable Deep Research and give it links to existing workout programs, articles from Men's Health, detail your current workout plan, goals, previous injuries, and and and and...<p>right...you can claim "everyone knows" about hallucinations. everyone knows they need to do prompt engineering. everyone knows how important hooking up an MCP server is. and if you use an LLM without knowing that, it's <i>your fault</i> for using the tool wrong.<p>but we know empirically that just isn't true. back in 2023, lawyers were sanctioned for filing LLM-generated legal briefs with hallucinated case citations [0]. then more than a year later, it happened again at a different law firm [1].<p>and does anyone think, realistically, that's the very last time we've seen a lawyer get sanctioned for LLM-hallucinated case citations?<p>the "dismissive" attitude towards LLMs that I find most compelling is that the benefits have been hugely oversold, and the caveats downplayed. LLMs have been rolled out in <i>way</i> too many places where they weren't ready for prime-time yet, and to users who weren't aware of the pitfalls.<p>Google has LLM-generated "overviews" at the top of every search result page. it famously said you could put glue on your pizza [2] and that it was safe to eat one rock per day [3].<p>you can say "everyone who uses an LLM <i>should</i> know such-and-such caveats" if you want. but Google's rollout of AI overviews mean that <i>every single person who uses Google Search</i> is suddenly an "LLM user". expecting all of those people to learn overnight how to use an LLM "properly" is simply not realistic.<p>0: <a href="https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/" rel="nofollow">https://www.reuters.com/legal/new-york-lawyers-sanctioned-us...</a><p>1: <a href="https://www.theregister.com/2025/02/25/fine_sought_ai_filing_mistakes/" rel="nofollow">https://www.theregister.com/2025/02/25/fine_sought_ai_filing...</a><p>2: <a href="https://www.theverge.com/2024/5/23/24162896/google-ai-overview-hallucinations-glue-in-pizza" rel="nofollow">https://www.theverge.com/2024/5/23/24162896/google-ai-overvi...</a><p>3: <a href="https://www.bbc.com/news/articles/cd11gzejgz4o" rel="nofollow">https://www.bbc.com/news/articles/cd11gzejgz4o</a>