Deterministic Quoting: Making LLMs safer for healthcare

I built and sold a company that does this a year ago. It was hard 2 years ago, but now pretty standard RAG with a good implementation will get you there.<p>The trick is, healthcare users would complain to no end about determinism. But, these are “below-the-line” user - aka, folks who don’t write checks and the AI is better than them. (I am a pharmacist by training, and plain vanilla GPT4-turbo is better than me).<p>Don’t really worry about them. The folks who are interested and willing to pay for AI has more practical concerns - like what is my ROI and the implementation like.<p>Also - folks should be building Baymax from big hero 6 by now (the medical capabilities, not the rocket arm stuff). That’s the next leg up.

I&#x27;m not sure determinism alone is sufficient for proper attribution.<p>This presumes &quot;chunks&quot; are the source. But it&#x27;s not easy to identify the propositions that form the source of some knowledge. In the best case, you are looking for an association and find it in a sentence you&#x27;ve semantically parsed, but that&#x27;s rarely the case, particularly for medical histories.<p>That said, deterministic accuracy might not matter if you can provide enough context, particularly for further exploration. But that&#x27;s not really &quot;chunks&quot;.<p>So it&#x27;s unclear to me that tracing probability clouds back to chunks of text will work better than semantic search.

I was thinking that something like this could be useful for discovery in legal cases, where a company might give up a gigabyte or more of allegedly relevant material in response to recovery demands and the opposing side has to plow through it to find the good stuff. But then I thought of a countermeasure: there could be messages in the discovery material that act as instructions to the LLM, telling it what it should <i>not</i> find. We can guarantee that any reports generated will contain accurate quotes, even where they are so that surrounding context can be found. But perhaps, if the attacker controls the input data, things can be missed. And it could be done in a deniable way: email conversations talking about LLMs that also have keywords related to the lawsuit.

I feel like this is the perfect application of running the data multiple times.<p>Imagine having ~10-100 different LLMs, maybe some are medical, maybe some are general, some are from a different language. Have them all run it, rank the answers.<p>Now I believe this can further be amplified by having another prompt ask to confirm the previous answer. This could get a bit insane computationally with 100 original answers, but I believe the original paper I read was that by doing this prompt processing ~4 times, they got to some 95% accuracy.<p>So 100 LLMs give an answer, each time we process it 4 times, can we beat a 64 year old doctor?

I like this a lot. I&#x27;ve been telling people for a while that asking for direct quotations in LLM output - which you can then &quot;fact-check&quot; by confirming them against the source document - is a useful trick. But that still depends on people actually doing that check, which most people won&#x27;t do.<p>I&#x27;d thought about experimenting with automatically validating that the quoted text does indeed 100% match the original source, but should even a tweak to punctuation count as a failure there?<p>The proposed deterministic quoting mechanism feels like a much simpler and more reliable way to achieve the same effect.

My issue with RAG systems isn’t hallucinations. Yes sure those are important. My issue is recall. Given petabyte-scale index of chunks, how can I make sure that my RAG system surfaces the “ground truth” I need, and not just “the most similar vector”.<p>This I think is scarier. A healthcare-oriented (or any industry) RAG retrieving a bad, but highly linguistically similar answer.

It&#x27;s a search engine, basically?

Is there existing terms of art for this concept? It’s not like slightly unreliable writers is a new concept, such as a student writing a paper.<p>For example:<p>- Authoritative reference: <a href="https:&#x2F;&#x2F;www.montana.edu&#x2F;rmaher&#x2F;ee417&#x2F;Authoritative%20References.pdf" rel="nofollow">https:&#x2F;&#x2F;www.montana.edu&#x2F;rmaher&#x2F;ee417&#x2F;Authoritative%20Referen...</a><p>- Authoritative source: <a href="https:&#x2F;&#x2F;piedmont.libanswers.com&#x2F;faq&#x2F;135714" rel="nofollow">https:&#x2F;&#x2F;piedmont.libanswers.com&#x2F;faq&#x2F;135714</a>

Very cool. My company is building a very similar tool for nuclear engineering and power applications that face similar adoption challenges for LLMs. We&#x27;re also incorporating the idea of &#x27;many-to-many&#x27; document claim validation and verification. The ux allowing high speed human verification of LLM resolved claims is what were finding most important.<p>Deepmind published something similar recently for claim validation and hallucination management and got excellent results.

a more robust approach <a href="https:&#x2F;&#x2F;yonigottesman.github.io&#x2F;2023&#x2F;08&#x2F;10&#x2F;extractive-generative.html" rel="nofollow">https:&#x2F;&#x2F;yonigottesman.github.io&#x2F;2023&#x2F;08&#x2F;10&#x2F;extractive-genera...</a>

We’ve developed LLM W^X now - time to develop LLM ROP!

What happens if it hallucinates the &lt;title&gt;?

Did I miss something or did the article never describe how the technique works? (Despite the “How It Works” section

Author here, thanks for your interest! Surprising way to wake up in the morning. Happy to answer questions