We don't need a string type (2013)

Curious. I have to come to exactly the opposite conclusion — that we should drop the idea of a fixed-length character type, and instead _only_ have (Unicode) string types. Actually, I&#x27;d prefer something like `std::text` to finally be free of the baggage of &quot;string&quot;. Operations on text should work on logical text concepts. For example, something like `someText.firstCharacter()` would have a return type of `text`, with logical length 1. It&#x27;s _data_ length is variable, since a Unicode character is variable length. So many Unicode-containing string design problems arise because of the stubborn insistence of having an integral character type.<p>I should be able to extract UTF-8, UTF-16 or whatever encoding I want from a `text` value. Something like `c_str()` would be pretty important, but the semantics would be a design problem, not an encoding problem. Any Unicode-encoding string should be able to encode U+0000, so you&#x27;d need to figure out how to handle that from `c_str()` (perhaps a substitution ASCII character could be specified to encode embedded nulls).<p>Basically, users should definitely _not_ need to understand the deeper details of Unicode. They shouldn&#x27;t need to understand and worry about different entities such as code units, code points, graphemes, and the like, though they should be able to extract such encodings on demand.

Go&#x27;s immutable UTF-8 string type is one of the nice things about the language<p>A Go string is almost exactly like this C struct:<p><pre><code> struct String { uint8_t* addr; ptrdiff_t len; }; </code></pre> The language guarantees you can&#x27;t modify the bytes in memory range [addr, addr+len)<p>Go&#x27;s garbage collection makes it simple and natural to have one string alias (&quot;point into&quot;, &quot;overlap&quot;) part of another string. This works because strings are immutable. Compare this to the nightmare in C++, where substrings require copying or explicit handling<p>The rune (UTF-8) iterator and other facilities make Unicode handling natural in Go<p>In summary, Go&#x27;s string type is a huge win

I think the problem is that, a lot of time when we deal with strings, we are thinking about ASCII strings instead of other encoding like UTF-8. If we treat them as ASCII strings, an array of characters would make sense, but it is not that simple for other encoding.<p>One of the languages that considered the issue is Rust. In rust, we don&#x27;t really index into strings, but use iterators or other methods to do the operations required. <a href="https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;string&#x2F;struct.String.html" rel="nofollow">https:&#x2F;&#x2F;doc.rust-lang.org&#x2F;std&#x2F;string&#x2F;struct.String.html</a>

The date should be (2013) not (2018), as that dates it before Rust 1.0 (which does have a UTF-8 string type) and before the Julia 1.0 release date (which implements UTF-8 strings as arrays with irregularly spaced indexes, eg, the valid indexes may be 1, 2, 4, 5, if the character at 2 takes up two bytes). Both would be interesting examples to compare against if this article was written today.

I think the author started from an assertion (&quot;This primary difference between a C++ ‘string’ and ‘vector’ is really just a historical oddity that many programs don’t even need anymore&quot;) that highlights an error in the C++ model of strings, not in the way we must think about strings.<p>Contrast NSString in Cocoa (<a href="https:&#x2F;&#x2F;developer.apple.com&#x2F;documentation&#x2F;foundation&#x2F;nsstring" rel="nofollow">https:&#x2F;&#x2F;developer.apple.com&#x2F;documentation&#x2F;foundation&#x2F;nsstrin...</a>). The Cocoa string is extremely opaque; it&#x27;s basically an object. And under the hood, that opacity allows for piles of optimization that are unsafe if the developer is allowed to treat the thing as just a vector of bytes or codepoints. Under the hood, Cocoa does all kinds of fanciness to the memory representation of the string (automatically building and cutting cords, &quot;interning&quot; short strings so that multiple copies of the string are just pointers to the same memory, caching of some transforms under the assumption that if it&#x27;s needed once, it&#x27;s often needed again).<p>Taken this way, one can even start to talk about things like &quot;Why does &#x27;indexing&#x27; into a string always return a character, instead of, say, a word?&quot; and other questions that are harder to get into if one assumes a string is just &#x27;vector of characters&#x27; or &#x27;vector of bytes.&#x27;

The article is an argument against types, in general.<p>The point that characters can be stored in other containers is meaningless: the question is whether, conceptually, a specific sequence of character values distinct from another sequence has compile-time meaning. It does. Therefore, it needs a type.<p>Such a sequence has numerous special characteristics. In particular, element at [i] often has an essential connection to element at [i+1] such that swapping them could turn a valid string to an invalid one. In fact, that an invalid sequence is even possible is another such characteristic.

String should be an interface&#x2F;protocol. When I log a message, I want to pass a string. If I have to append large strings for a log message I don&#x27;t want to run out of memory, I should be able to pass a rope&#x2F;cord [1]. We&#x27;ve known how to abstract this for forever and should work to optimize our compilers&#x2F;runtimes accordingly. I&#x27;m not aware of a language which has got this right, for example, Java has the ugly CharSequence interface that nobody uses. StringProtocol in Swift (can I implement it?) makes you pay a character tax rather than to just pass a string. Rust&#x2F;C++ give various non-abstracted types.<p>[1] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Rope_(data_structure)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Rope_(data_structure)</a>

I think that the problem with text is that the basic operation you want to do is inserts. The way memory works in computer makes that an inherently inefficient operation. I&#x27;m a bit fascinated by how bad computer are at text give that that is what we use so much of them for.<p>As a C programmer I think that its not really possible to implement an efficient text processing library, because there is no good universal way to store text. So much depends on the pattern of the processing functions. If you want to avoid allocating new memory and moving a lot of text for each operation, the implementation needs to make speculative choices about how text can best be stored. How you store text depends so much on your access pattern. Do you need to be able to get to a line fast? or know how long the text is? Or insert something? and if so how much?<p>A C style string would for instance be terrible for something like a text editor, because every key press would cause a complete copy of the document to have to be allocated, and then copied over. So maybe a linked list? But you dont want just one character in each link because that trashes the cache right? but then its still slow to just skip forward fast, so maybe an array of pointers to snipets? or maybe a linked list of pointers to snippets? So many possibilities that all impact performance differently depending on what you do with it.<p>When I see higher languages with nice easy to use string functionality, I always consider, the impossible choices that had to be made under the hood.

Surprising to a Tcl programmer!8-)) b&#x2F;c<p>&quot;Everything is a String&quot;:<p><a href="https:&#x2F;&#x2F;wiki.tcl-lang.org&#x2F;page&#x2F;everything+is+a+string" rel="nofollow">https:&#x2F;&#x2F;wiki.tcl-lang.org&#x2F;page&#x2F;everything+is+a+string</a><p>and<p>&quot;Everything is a Symbol&quot;:<p><a href="https:&#x2F;&#x2F;wiki.tcl-lang.org&#x2F;page&#x2F;Everything+is+a+Symbol" rel="nofollow">https:&#x2F;&#x2F;wiki.tcl-lang.org&#x2F;page&#x2F;Everything+is+a+Symbol</a>

The author has these followup blogposts :<p>2013 : <a href="https:&#x2F;&#x2F;mortoray.com&#x2F;2013&#x2F;11&#x2F;27&#x2F;the-string-type-is-broken&#x2F;" rel="nofollow">https:&#x2F;&#x2F;mortoray.com&#x2F;2013&#x2F;11&#x2F;27&#x2F;the-string-type-is-broken&#x2F;</a><p>2014 : <a href="https:&#x2F;&#x2F;mortoray.com&#x2F;2014&#x2F;03&#x2F;17&#x2F;strings-and-text-are-not-the-same&#x2F;" rel="nofollow">https:&#x2F;&#x2F;mortoray.com&#x2F;2014&#x2F;03&#x2F;17&#x2F;strings-and-text-are-not-the...</a><p>(See also : <a href="https:&#x2F;&#x2F;thehardcorecoder.com&#x2F;2014&#x2F;04&#x2F;15&#x2F;data-text-and-strings-oh-my&#x2F;" rel="nofollow">https:&#x2F;&#x2F;thehardcorecoder.com&#x2F;2014&#x2F;04&#x2F;15&#x2F;data-text-and-string...</a> )<p>2016 : <a href="https:&#x2F;&#x2F;mortoray.com&#x2F;2016&#x2F;04&#x2F;28&#x2F;what-is-the-length-of-a-string-a-tricky-question&#x2F;" rel="nofollow">https:&#x2F;&#x2F;mortoray.com&#x2F;2016&#x2F;04&#x2F;28&#x2F;what-is-the-length-of-a-stri...</a>

Discussed at the time: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=6204427" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=6204427</a>

I can&#x27;t speak for C++, but for C, the repeated issue is that a null-terminated string has lots of utility routines that are handy for manipulating them. Without 3rd party libraries, plain length-header buffers don&#x27;t. Hence things like Antirez&#x27;s sds library, which by nature, is a compromise. I get you can&#x27;t fundamentally change C now, but a buffer type with a rich manipulation library would have been nice.

Anyone else thinks that we missed an opportunity to make text much simpler to deal with by not increasing the size of a byte from 8 to 32 bits when we moved from 32-bit to 64-bit word length CPUs ?<p>I mean, isn&#x27;t the 7-bit ASCII text the reason why the byte length was standardized to the next power of two bits ?<p>(With e-mail still supporting non-padded 7-bit ASCII until recently for performance reasons.)

TL;DR : Characters and Strings considered harmful.<p>And he&#x27;s right, they totally are ! (Also, &#x27;string&#x27; can mean an ordered sequence of similar objects of any kind, not just characters.)<p>But (as these discussions also mention) replacing them by much more clearly defined concepts like byte arrays, codepoints, glyphs, grapheme clusters and text fields is only the first step...<p>The big question (these days) is what to do with text, specifically the &#x27;code&#x27; kind of text (either programming or markup, and poor separation between &#x27;plain&#x27; text and code keeps causing security issues).<p>To start with, even code needs formatting, specifically some way to signal a new line, or it will end up unreadable.<p>Then, code can&#x27;t be just arbitrary Unicode text, some limits have to apply, because Unicode can get verrrry &#x27;fancy&#x27; ! (Arbitrary Unicode is fine in text fields and comments embedded in code.)<p>So, I&#x27;m curious, is there any Unicode normalization specifically designed for code ? (If not, why, and which is the closest one ?)<p>I&#x27;m thinking of Python (3), which has what seems to be a somewhat arbitrary list of what can and what can&#x27;t be used as a variable name ? (And the language itself seemingly only uses ASCII, though this shouldn&#x27;t be a restriction for programming&#x2F;markup languages !)<p>Also I hear that Julia goes much further than that (with even (La)TeX-like shortcuts for characters that might not be available on some keyboards), what kind of &#x27;normalization&#x27; have they adopted ?