Understanding Surrogate Pairs: Why Some Windows Filenames Can't Be Read

The script works just fine on real Linux, it creates 2048 files and <i>ls</i> command lists them all with different names.<p><pre><code> ls -l win32&#x2F; total 0 -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\277\237&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\267\213&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\240\220&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\274\273&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\251\205&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\255\223&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\272\257&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\264\207&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\261\246&#x27;&#x27;.exe&#x27; -rw-r--r-- 1 dawid dawid 0 Feb 26 12:13 &#x27;&#x27;$&#x27;\355\254\266&#x27;&#x27;.exe&#x27; ...</code></pre>

Falsehoods programmers believe about filenames #1: Filenames are text and can be represented in common text encodings.<p>&gt; Windows was an early adopter of Unicode, and its file APIs use UTF‑16 internally since Windows 2000<p>Wrong. Windows uses WTF-16 [0] despite what the documentation says.<p>[0] <a href="https:&#x2F;&#x2F;simonsapin.github.io&#x2F;wtf-8&#x2F;#ill-formed-utf-16" rel="nofollow">https:&#x2F;&#x2F;simonsapin.github.io&#x2F;wtf-8&#x2F;#ill-formed-utf-16</a>

Hi all. OP here. I added a Postscriptum about the surrogte pairs and their status in Linux. I used WSL to access those files under Windows, and generated the same on Linux. You can see that behavior differs on the same file names:<p>1. On Windows, accessed by WSL<p>2. On Linux (WSL), using UTF-8 locale<p>3. On Linux (WSL), using POSIX locale<p>The difference is weird for me as a user. I&#x27;d like to know about the decisions made behind these. If anyone has information, please let me know.

I remember that in Mac OS X times, sometime between OS X v10.1 and 10.4, a system upgrade caused a bunch of unicode named files to become inaccessible&#x2F;untouchable (but still present with a directory listing). At the time I didn&#x27;t have the skills to figure out what had happened. I&#x27;m still curious to know if it was an intended breaking change.

Hi, thanks for the interesting submission!<p>I was a bit confused by the detour via utf-8 to arrive at the code points and had to look up UTF-8 encoding first to understand how they relate. Then I tried out the following<p><pre><code> candidate = chr(0xD800) candidate2 = bytes([0xED, 0xA0, 0x80]).decode(&#x27;utf-8&#x27;, errors=&#x27;surrogatepass&#x27;) print(candidate == candidate2) # True </code></pre> and it seems that you could just iterate over code points directly with the `chr()` function.

Aha! Found the name of my next album. Try downloading me on Napster now!

Ah got caught by surrogate pairs recently:- javascript sees them as 2 chars when e.g. slicing strings, so it is possible to end up with invalid strings if you chop between a pair.

I think it&#x27;s hilarious that the event viewer XML gets borked.

Why does the Windows filesystem allow filenames with invalid strings?<p>It seems obvious that attempts to create files with such filenames ought to be blocked.

I remember a long time ago I accidentally put some symbol in a folder name like ? in Windows had problems

Or otherwise said: Surrogate Pairs are used in UTF-16 (which uses two bytes per character, so it can encode up to 65536 characters) to encode Unicode characters that have code points that can&#x27;t be encoded using just two bytes.

Stuff like this is why UTF and any attempt at trying to encode all characters is a mistake.<p>The real solution is to force the entire world population to use the Rotokas language of Papua New Guinea.

Yet another reason to use Linux everywhere. It is 2025 and Windows (and probably Mac) users have to deal with weird Unicode filesystem issues. Good luck puting Chinese characters or emoticons into filenames.<p>Ext4 filename has maximal length 255 characters. That is the only legacy limit you have to deal with as a Linux user. And even that can be avoided by using more modern filesystems.<p>And we get filesystem level snapshots etc...