ChatML: ChatGPT API expects a structured format, called Chat Markup Language

(I work at OpenAI.)<p>This document is a preview of the underlying format consumed by ChatGPT models. As an API user, today you use our higher-level API (<a href="https:&#x2F;&#x2F;platform.openai.com&#x2F;docs&#x2F;guides&#x2F;chat" rel="nofollow">https:&#x2F;&#x2F;platform.openai.com&#x2F;docs&#x2F;guides&#x2F;chat</a>). We&#x27;ll be opening up direct access to this format in the future, and want to give people visibility into what&#x27;s going on under the hood in the meanwhile!

Is it just me or is this the least intuitive format imaginable? The type def is something like:<p><pre><code> type Message = string type Speaker = &#x27;system&#x27; | &#x27;user&#x27; | &#x27;assistant&#x27; | &#x27;system name=example_user&#x27; | &#x27;system name=example_assistant&#x27; type CML = (&#x27;\n&#x27; | &#x27;${Speaker}\n${Message}&#x27; | {token: &#x27;&lt;im_start&gt;&#x27;|&#x27;&lt;im_end&gt;&#x27;})[] </code></pre> I&#x27;d expect something more like...<p><pre><code> type Message = string type Speaker = &#x27;system&#x27; | &#x27;user&#x27; | &#x27;assistant&#x27; | &#x27;example_user&#x27; | &#x27;example_assistant&#x27; type CML = {message: Message, speaker: Speaker}[]</code></pre>

Why do you think Bing Chat is going in a slightly different direction and not using this format exactly?<p>Their prompt - <a href="https:&#x2F;&#x2F;old.reddit.com&#x2F;r&#x2F;bing&#x2F;comments&#x2F;11bd91j&#x2F;release_of_the_whole_initial_prompt_of_bing_chat&#x2F;" rel="nofollow">https:&#x2F;&#x2F;old.reddit.com&#x2F;r&#x2F;bing&#x2F;comments&#x2F;11bd91j&#x2F;release_of_th...</a><p>Wouldn&#x27;t it be better to unify around ChatML, for the sake of all future training data being consistent? I thought it was strange that they used &lt;|im_start|&gt; but not the rest of the ChatML syntax.<p>(There is always the possibility that they <i>are</i> using it, but the AI has hallucinated a slightly different syntax when repeating it)

Here is a leaked google doc with a lot more details: <a href="https:&#x2F;&#x2F;docs.google.com&#x2F;document&#x2F;d&#x2F;1mYBAIilR8IcIfzvIfrsayAU_XJJ-w5Oi6zYY53g0LFs&#x2F;edit?usp=sharing" rel="nofollow">https:&#x2F;&#x2F;docs.google.com&#x2F;document&#x2F;d&#x2F;1mYBAIilR8IcIfzvIfrsayAU_...</a><p>This was copied from OpenAI alpha documentation.

In what way is this a “list of dicts”? It’s an implicitly structured list of dict, str pairs. And it is WEIRD.

I just rewired our project from &lt;|im_start|&gt;&lt;|im_end|&gt; to use the { &quot;role&quot; : &quot;user&quot;, &quot;content&quot; : &quot;Hi!&quot; } format.<p>Naming-wise, the JSON format is not ChatML, right? Does it have a name yet?<p>(This is what I&#x27;m working off of: <a href="https:&#x2F;&#x2F;platform.openai.com&#x2F;docs&#x2F;api-reference&#x2F;chat&#x2F;create" rel="nofollow">https:&#x2F;&#x2F;platform.openai.com&#x2F;docs&#x2F;api-reference&#x2F;chat&#x2F;create</a>)

ChatML documents consists of a sequence of messages. Each message contains a header and contents. The current version (ChatML v0) can be represented with a JSON format.

huh neat. a couple weeks back I had a conversation with ChatGPT where I asked it if it had special control tokens it could emit, and after a lot of coaxing I got it to tell me about &lt;|im_end|&gt;. I wasn&#x27;t sure if it hallucinated it but I guess not?

Do you have to prefix every user query with a long list of examples to create context (and be billed for it extra for every query)?<p>Or there is a way to create context with some prelude and then use it for subsequent queries?<p>Let&#x27;s say I want to create quick help for SQL, where the prelude would be schema snapshot and some examples.<p>Do I need to flood every user query with this long prefix of sql schema snapshot with examples?<p>I don&#x27;t want one user conversatio to interfere with other user query.

What is the best way for a response to come back simply as an integer? So asking, what is the avg. cost of a bagel in new york? I&#x27;ve been able to do it by asking for no text - but wondering if there is an alternative i&#x27;m missing.

Amusingly when I asked bing about this format hallucinated that there was also extra tokens such as &lt;|mood|&gt; to set style and tome. Take note @gdb :-D

I wonder why they are not using XML.

What is the plan to solve injections using this lower level representation?

Can I use ChatGPT to rewrite everything to this format?

It is worth discussing whether the domain specific language design for ChartML is necessary for users.

What does &quot;im&quot; stand for?

&gt; This gives an opportunity to mitigate and eventually solve injections<p>No more jailbreaks!!!!