The Turkish İ Problem and Why You Should Care (2012)

German has an &#x27;ß&#x27; problem of a similar nature. There is a corresponding capital &quot;ẞ&quot; in Unicode, and Germany has officially adopted &#x27;ẞ&#x27; as an alternative since, but in Unicode&#x27;s SpecialCasing.txt the upper of &#x27;ß&#x27; is still &#x27;SS&#x27;. The lower of &#x27;S&#x27; of course being &#x27;s&#x27;, there&#x27;s no going back after folding to upper cases. Lower of &#x27;ẞ&#x27; is however still &#x27;ß&#x27;.<p>So by alternating case you end up with ß→SS→ss or ẞ→ß→SS. Certainly has potential to screw with naive attempts at case-insensitive comparison via case folding. Then again, Unicode adopting &#x27;ẞ&#x27; as the upper of &#x27;ß&#x27; in some future version would probably only increase that potential further.<p>I&#x27;m interested to hear from people dealing with a lot of German text how much of a problem this is in practice.

Transliterating this character incorrectly resulted in a violent attack causing two deaths:<p><a href="https:&#x2F;&#x2F;languagelog.ldc.upenn.edu&#x2F;nll&#x2F;?p=73" rel="nofollow">https:&#x2F;&#x2F;languagelog.ldc.upenn.edu&#x2F;nll&#x2F;?p=73</a>

&gt; So while we have two i’s (upper and lower), they have four.<p>No, we don’t have four i’s in Turkish. I(ı) and İ(i) are two separate letters. Turkish alphabet has 29 letters, and each letter has their own key on a Turkish keyboard. We also have Öö, Üü, Çç, Şş and Ğğ. These are all individual letters. They are not dotted versions, accents or a writing convention. So the language is as simple as it gets. The complications come from mapping letters to English alphabet.

I feel like Turkish should have been given a different entirely separate lowercase &quot;i&quot; character so the pairs could be consistent, like the Greek lookalikes. Considering how historically capital letters came before lowercase it seems like İ should have been considered an entirely separate letter from I.<p>Greek was given entirely separate characters even though many are indistinguishable from the Latin alphabet. In Greek for instance we have &quot;Ν&quot; lowercases to &quot;ν&quot; instead of &quot;n&quot;. The Greek &quot;Ν&quot; however is not a latin &quot;N&quot; and is an entirely separate character. This makes a lot more sense.

This is one of the reasons why software development is so difficult, most people cannot even begin to imagine how complex the user environment can be. Even within very niche problem domains you may have to deal with a broad range of different environments with different locales, different spoken languages, operating systems, programming languages, compilers&#x2F;transpilers, engine versions, server frameworks, cache engines, load balancers, TLS certificate provisioning, container engines, container image versions, container orchestrators, browsers, browser extensions, frontend frameworks, test environments, transfer protocols, databases (with different client and servers versions), database indexes, schema constraints, rate limiting... I could probably keep going for hours. Now imagine being aware of all these factors (and much more) and being aware of all possible permutations of these; that&#x27;s what you need in order to be a senior software developer these days. It&#x27;s a miracle that any human being can produce any working software at all.<p>As a developer, if some code works perfectly on your own computer, the journey has barely just begun.

This makes me wonder is a there a programming language which has separate data types for locale aware and locale independent strings. I know that rust has OsString but that&#x27;s a slightly different usecase.<p>Problem with the current widely used approach of having global application wide locale setting is that most applications contain mix of User facing strings and technical code interacting with file formats or remote APIs. Doesn&#x27;t matter if you set it to current language (or just let operating system set it) or force it to language independent locale, sooner or later something is going to break.<p>If you are lucky a programming language might provide some locale independent string functions, but using them is often clunky and and unlikely to be done consistently across whole code base and all the third party libraries. It&#x27;s easier to do things correctly if you are forced to declare the intention from the start and any mixing of different context requires an explicit conversion.

I think the key to doing text sanely in programming is separating &quot;text&quot; from &quot;international text&quot; or &quot;user text&quot;. &quot;Text&quot;, can be e.g. the characters that make up my xml node names. Or all the names of my db columns etc. You still have to worry about encodings and everything with this data, but you don&#x27;t have to worry that there is a 10 byte emoji or a turkish upper case i. A key property of it is: you can, for example, run toUpper or toLower, with a default culture. It has symmetric transforms. It can often be assumed to be the ASCII subset, regardless off encoding.<p>Then on the other end you have text what the user enters. It can be anything (so may need validation and washing). You may not be able to run &quot;to lower&quot; on it (although I&#x27;d be tempted to do it on an email address for example).<p>The key is just knowing what you have. It&#x27;s unfortunate that &quot;string&quot; is usually used for everything from paths to user input to db column names etc.

System.Globalization is quite the feat of engineering. Setting CultureInfo is like getting onto an actual airplane. I don&#x27;t know of any other ecosystem with docs like:<p><a href="https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;windows&#x2F;apps&#x2F;design&#x2F;globalizing&#x2F;japanese-era-change" rel="nofollow">https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;windows&#x2F;apps&#x2F;design&#x2F;global...</a>

Boy it would sure be easier if the Turkish i was a different unicode character in lowercase too

Ahh yes, been there, done that.<p>Several years ago we had issues with certification of our game on PS4 because the capitalization on Sony&#x27;s Turkish translation for &quot;wireless controller&quot; was wrong. The problem being that Turkish dotless I. What was the cause? Some years prior we had had issues with internal system strings (read: stringified enums) breaking on certain international PC&#x27;s because they were being upper&#x2F;lowercased using locale-specific capitalization rules. As a quick fix, the choice was made then to change the culture info to invariant globally across the entire game. This of course meant that <i>all</i> strings were now being upper&#x2F;lowercased according to English rules, including user-facing UI strings. Hence Turkish strings mixing up dotted and dotless I&#x27;s in several places. The solution? We just pre-uppercased that one &quot;wireless controller&quot; term in our localization sheet, because that was the only bit of text Sony cared about. An ugly fix and we really should have gone through the code to properly separate system strings from UI texts, but it got the job done.

Over the years this has shown up a few times because PHP internally was using a locale dependent function to normalize the class names, but it was also doing it inconsistently in a few places. The bug was active for years and has resurfaced more than once: <a href="https:&#x2F;&#x2F;bugs.php.net&#x2F;bug.php?id=18556" rel="nofollow">https:&#x2F;&#x2F;bugs.php.net&#x2F;bug.php?id=18556</a>

This highlights the single biggest problem I have with the MS&#x2F;C#&#x2F;.NET runtime&#x2F;ecosystem (The article seems to be from a .NET developer), so many functions connected to string handling are locale dependent and you have to explicitly select the non-locale variants and that then becomes an issue when dealing with the common data interchange and file-formats since those are usually with US semantics.<p>Many European developers run into this frequently since the default parse for float&#x2F;double&#x2F;decimal will assume comma as the decimal separator due to our locale settings.

I wish someone would write a book that distills all the knowledge contained in those &quot;Falsehoods Programmers Believe About X&quot; or &quot;Things Programmers Should Know&quot; topics, providing a resource for how to write real-world practically robust software that works reasonably well anywhere anytime.<p>The list of gotchas with any non-trivial software is long and frequently obscure.

Reminds me to friends old but brilliant project, use Unicode to draw art on stack trace logs! Enough with boring stack traces in logs, lets make some art there and make life a bit easier for the poor soul thats on support and has to debug latest prod issue. <a href="https:&#x2F;&#x2F;medium.com&#x2F;@ironcev&#x2F;stack-trace-art-4b700a8817ea" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;@ironcev&#x2F;stack-trace-art-4b700a8817ea</a>

When I was in Turkey on a project, the i was absolutely a problem in the software I was trying to deploy. Glad to see this as it&#x27;s one of those classic &quot;Things Programmers Should Know&quot; topics right up there with all the other classics like address formats and name formats not being the same across the globe.

Huh. Trying to find the letter &quot;i&quot; in this page in Firefox for Android results in a 0-based index of results (starts at 0&#x2F;-1); you get 999&#x2F;-1 as the last result if you start from the end.

We need a combining character DELETE DOT ABOVE to make i into ı.

<p><pre><code> const string input = &quot;interesting&quot;; bool comparison = input.ToUpper() == &quot;INTERESTING&quot;; Console.WriteLine(&quot;These things are equal: &quot; + comparison); Console.ReadLine(); </code></pre> Is this a realistic scenario? Changing case of a string and comparing it to something else? Running some kind of operations &amp; logic on a string that is meant for user?<p>If you are doing such things then it looks more like a code smell.