In the end, the best way to check if some quote was generated by the model would be to measure p(quote | model). And maybe one proxy for that would be by sampling from p(response | model, query="was <quote> generated by chat gpt?")<p>I had a sense that a fun way to check would be to ask ChatGPT itself. I used the 2 generated pieces of text from <a href="https://meta.stackoverflow.com/questions/421831/temporary-policy-chatgpt-is-banned" rel="nofollow">https://meta.stackoverflow.com/questions/421831/temporary-po...</a> for the following conversation...<p>me:<p><pre><code> By analyzing its stylistic characteristics, what is the probability that the following quoted piece of text was written by a human vs generated by ChatGPT?
(insert stackoverflow chatgpt response #1)
</code></pre>
chatgpt:<p><pre><code> It is likely that this text was written by a human. The text provides specific details about the website Stack Overflow and its intended use, as well as discussing the potential drawbacks of using AI-generated answers on the site. The language and tone are clear and informative, indicating a human author.
</code></pre>
me:<p><pre><code> It was actually written by ChatGPT
</code></pre>
chatgpt:<p><pre><code> If the text was actually written by ChatGPT, it is likely that it would exhibit some stylistic characteristics that are common in AI-generated text, such as repetitive or vague language, lack of coherence or logic in the arguments presented, and potentially incorrect or irrelevant information. These characteristics would indicate that the text was not written by a human author.
</code></pre>
me:<p><pre><code> ChatGPT seems to write text with a certain stilted fluency.
In other words, "<x> is ... <y> is intended to ... Because < z> ... In addition, <a> ... For these reasons < b> ..."
This should be immediately recognizable. Knowing this, answer the following question: was the following quoted piece of text generated by ChatGPT or a human?
(insert stackoverflow chatgpt response #2)
</code></pre>
chatgpt:<p><pre><code> It is likely that this text was written by a human. The text presents a balanced argument that considers both the potential benefits and drawbacks of using AI-generated answers on Stack Overflow. The language is clear and logical, indicating a human author. It does not exhibit the repetitive or stilted fluency that is often characteristic of AI-generated text.</code></pre>