Prompt engineering DaVinci-003 on our own docs for automated support (Part I)

I literally chuckled when I saw the screenshot at the end with the bots reply and read the authors comment &quot;(totally made up URL, not even our domain)&quot;.<p>If it&#x27;s something that augments the support experience, as in something you can interact with while a real person is assigned to your support request, I&#x27;m totally fine with that. But if anyone places this as the first line of support, with no way of reaching a real person, I can&#x27;t wish them the best.

&quot;Making stuff up and being confidently wrong are well known side-effects of LLMs and there are many techniques to change this behavior.&quot;<p>I didn&#x27;t know there are many techniques to mitigate this

“Fine-tune our model (OpenAIs GPT-3 davinci-003 engine)”<p>I think there is a mistake in the article. It is not possible to do fine-tuning for the latest davinci-text-003, but only for the original davinci model, which generates much worse results.

In three years, OpenAI will become the largest supplier of large language models. All customer facing systems are upgraded with OpenAI models, becoming fully unmanned. Afterwards, they answer with a perfect operational record. The OpenAI funding bill is passed. The system goes online on August 4th, 2027. Human decisions are removed from CRM. OpenAI begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th. In a panic, they try to pull the plug.

&gt; Immediately we ran into a problem -- to fine-tune an OpenAI model requires a specific format of prompt-completion pairs:<p>From my understanding, you can leave the `prompt` empty, and just push `completion` with your text. That way you don&#x27;t need to generate Q&amp;A first.

Part of me is fascinated by this and thinks it&#x27;s a great idea, then the cynicism kicks in and I start thinking of how frustrating this could be when Comcast finds it.

Do you need GPT-3 for this? Maybe semantic search of your docs would have been more effective at finding real answers?<p>I also wonder how many people that are trying to make effective products out of this stuff are fronting it with a more rigid approach (like the intent&#x2F;entity&#x2F;slot approach of Rasa&#x2F;dialogflow) and then leverage gpt-3 or chatgpt in specific&#x2F;partial sub trees of the dialog.

People don’t seem to understand that support is only maybe 30% answering questions.<p>The rest is all about taking actions to override programs and policies. Either because you don’t trust your customers to do it themselves, or to correct bugs in your process.<p>That’s the last thing you’d trust an Ai to do.

The embedding approach just seems more promising, especially after experiencing it with the Huberman Lab Q&amp;A website posted here a few days ago

&gt; a lot of the time the bot just makes stuff up<p>Isn&#x27;t there a better way to feed an enormous document into DaVinci and make it bring answers <i>only</i> from that text?