GPT-4o's Chinese token-training data is polluted by spam and porn websites

&gt; The new tokenizer has 200,000 tokens in total, and about 25% are in non-English languages.<p>You have to hand it to the British: of all the different peoples and nations who tried their hand at world domination, the tiny little island of Britain won.

pure curiosity, haha, just run it (<a href="https:&#x2F;&#x2F;gist.github.com&#x2F;ctlllll&#x2F;4451e94f3b2ca415515f3ee369c8c374" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;ctlllll&#x2F;4451e94f3b2ca415515f3ee369c8...</a>) through translate to english:<p>here are the first tokens till ctrl+c ---<p>&quot;It&#x27;s a bit outrageous...&quot;<p>Japanese Adult Videos Free to Watch<p>China Welfare Lottery Daily<p>Long-lasting Free Hot Online Selections<p>Uncensored High Definition Free Videos<p>DaFa Fast Three - Big&#x2F;Small, Single&#x2F;Double<p>Leave Something for the Master<p>QQ&#x27;s Daily Lottery<p>Japanese Full-Length Adult Film<p>DaFa Fast Three Lottery Results<p>Invitation Code for CaiShen Competition<p>Free Videos Online Watch<p>Uncensored High Definition Free Videos<p>Uncensored Videos - First, Second, Third Zones<p>DaFa Timely Lottery Plan<p>DaFa Timely Lottery Results<p>DaFa Timely Lottery How-to<p>DaFa Fast Three by the State?

Aww. The data they stole isn’t ideal? Such a shame.

well, that&#x27;s a given, bro. you guys only steal yet you complain when what you steal makes your result &quot;poor&quot;.