The UTF-8-Everywhere Manifesto

Really good article. You'll get nothing from me but heartfelt agreement. I especially liked that the article was giving numbers about how inefficient UTF8 would be to store Asian text (not really apparently).<p>Also insightful, but obvious in hindsight: Not even in utf-32 you can index specific character in constant time due to the various digraphs.<p>The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string.<p>This is a really good help for detecting encoding errors very early (still to this day, applications are known to lie about the encoding of their output).<p>And of course, there's no endianness issue, removing the need for a BOM which makes it possible for tools that operate at byte levels to still do the right job.<p>If only it had better support outside of Unix.<p>For example, try opening a UTF8 encoded CSV file (using characters outside of ASCII of course) in Mac Excel (latest versions. Up until that, it didn't know UTF8 at all) for a WTF experience somewhere between comical and painful.<p>If there is one thing I could criticize about UTF8 then that would be its similarity to ASCII (which is also its greatest strength) causing many applications and APIs to boldly declare UTF8 compatibility when all they really can do is ASCII compatibility and emitting a mess (or blowing up) once they have to deal with code points outside that range.<p>I'm jokingly calling this US-UTF8 when I encounter it (all too often unfortunately), but maybe the proliferation of "cool" characters like what we recently got with Emoji is likely going to help with this over time.

Ok, let me be the first approving top level comment: This document is correct. The author of this document is smart. You should follow this document.<p>As jwz said about backups: "Shut up. I know things. You will listen to me. Do it anyway."

Yes! I have been meaning to write something like this for years.<p>There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files.<p>Edit: Here is the interesting story of how Ken Thompson invented UTF-8: <a href="http://doc.cat-v.org/bell_labs/utf-8_history" rel="nofollow">http://doc.cat-v.org/bell_labs/utf-8_history</a>

Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks.<p>For new APIs in which legacy interoperability isn't needed, I completely approve of this document.

Personally, I prefer UTF-8 as well. However, I think this whole debate about choice of encoding gets blown out of proportion.<p>Consider the following diagram:<p><pre><code> [user-perceived characters] &#60;-+ ^ | | | v | [characters] &#60;-&#62; [grapheme clusters] | ^ ^ | | | | v v | [bytes] &#60;-&#62; [codepoints] [glyphs] &#60;----------+ </code></pre> Choice of encoding only affects the conversion from bytes to codepoints, which is pretty straight-forward: The subtleties lie elsewhere...

Disagree<p>"UTF-16 is the worst of both worlds—variable length and too wide"<p>Really, the author tries to convince the reader, but it's not that clean cut.<p>One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32)<p>"It's not fixed width" But for most text, it is. Sure, you can do UTF-32 and it may not be a bad idea (today)<p>Yes, Windows has to deal with several complications and with backwards compatibility, so it's a bag of hurt. Still, they went the right way (internally, it's unicode, period.)<p>"in plain Windows edit control (until Vista), it takes two backspaces to delete a character which takes 4 bytes in UTF-16"<p>If I'm not mistaken this is by design. The 4 byte characters is usually typed as a combination of characters, so if you want to change the last part of the combination you jut type one backspace.

Markus Kuhn's web page has a lot of useful UTF-8 info and valuable links (e.g. samples of UTF-8 corner cases that people often miss).<p><a href="http://www.cl.cam.ac.uk/~mgk25/unicode.html" rel="nofollow">http://www.cl.cam.ac.uk/~mgk25/unicode.html</a>

Totally agree re: UTF-8 vs other Unicode encodings.<p>But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: <a href="http://en.wikipedia.org/wiki/Han_unification" rel="nofollow">http://en.wikipedia.org/wiki/Han_unification</a>

For those who don’t know it, UTF8-CPP[1] is a good lightweight header-only library for UTF conversions, mostly STL-compatible.<p>[1] <a href="http://utfcpp.sourceforge.net/" rel="nofollow">http://utfcpp.sourceforge.net/</a>

That collection of best practices can hardly be considered as "UTF-8 Everywhere Manifesto" as it focuses on Windows and C++. It's good, but I'd rather see more manifesto like document for all cases on a domain like that.

Text is maddening, the modern Tower of Babel.<p>Is there a definitive reference, or small handful of references, to learn all that's worth knowing about text, from ASCII to UTF-∞ and beyond?

I use UTF-8 for transmitted data and disk I/O, and I use UCS-4 (wchar_t on Linux/FreeBSD) for internal representation of strings in my software.<p>I generally agree with this article, but I disagree with it on the point that UTF-8 is the only appropriate encoding for strings stored in memory, and also I disagree on the point wchar_t should be removed from C++ standard or made sizeof 1, as in Android NDK.<p>Let me explain why.<p>In UTF-8 single Unicode character may be encoded in multiple ways. For example NUL (U+0000) can be encoded as 00 or as C0 80. The second encoding is illegal because it's longer than necessary and forbidden by standard, but naive parser may extract NUL out of it. If UTF-8 input was not properly sanitized, or there is a bug in charset converter, this may result in exploit like SQL injection or arbitrary filesystem access or something like that: malicious party can encode not only NUL, but ", /, \ etc this way.<p>Also UTF-8 string can't be cut at arbitrary position. Byte groups (UTF-8 runes) must be processed as a whole, so appear either on left side or on the right side of cut.<p>Reversing of UTF-8 string is tricky, especially when illegal character sequences are present in input string and corresponding code points (U+FFFD) must be preserved in output string.<p>I think UTF-8 for network transmitted data and disk I/O is inevitable, but our software should keep all in-memory strings in UCS-4 only, and take adequate security precautions in all places where conversion between UTF-8 and UCS-4 happens.<p>And sizeof(wchar_t)==4 in GCC ABI is not a design defect, wchar_t exists for a good reason. I admit that sizeof(wchar_t)==2 on Windows is utterly broken.

The strangest thing about Unicode (any flavor) is that NULL, aka \0, aka "all zeros" is a valid character.<p>If you claim to support Unicode, you have to support NULL characters; otherwise, you support a subset.<p>I find most OS utilities that "accept" Unicode fail to accept the NULL character.<p>FWIW, UTF-8 has a few invalid characters (characters that can never appear in a valid UTF-8 string). Any one of them could be used as an "end of string" terminator if so desired, for situations where the string length is not known up front.<p>We could even standardize which one (hint hint). I suggest -1 (all 1s).<p>UPDATE: I meant "strange" as in "surprising", especially for those coming from a C background, like me.

Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?

can not agree more! it will be a much better world if we all use utf8 for external string presentation. i don't care about what your app use internally, but if it generates output, please use utf8.

I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness?<p>In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

How could we avoid acronyms like 'utf-8'?<p>We can do better than that. Unicode8?

That page is misleading when it comes to Japanese text: UTF-8 sucks for Japanese text. UTF-8 and UTF-16 aren't the only two choices within the whole world, which is demonstrated in their choice of encoding Shift-JIS.

tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else.<p>UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better.<p>There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8.<p>C# and Java are probably the best examples of software that was engineered from the ground up and thus uses UTF-16 internally because it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters. If you use UTF-8 expect this result anytime a string contains a real genuine apostrophe.<p>"UTF-8 and UTF-32 result the same order when sorted lexicographically. UTF-16 does not."<p>This is complete and utter bullshit, to sort a string lexicographically you need to decode it, if you've decoded the string into UNICODE then they sort the exact same way.<p>There are lots of gotchas for sorting UNICODE strings including normalization because you can write the semantically equivalent strings in unicode multiple ways. eg. ligatures.<p>If you're sorting bit strings that happen to contain UTF-8/32 then you're not sorting lexicographically and your results will be screwed up anyway.

ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low.<p>Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?