Tree Sitter and the Complications of Parsing Languages

&gt; Well, because it’s gosh-darn hard to do it the right way.<p>I think this overstates the difficulty. This of course depends a lot on the language, but for a reasonable one (not C++) you can just go and write the parser by hand. I’d ballpark this as three weeks, if you know how the thing is supposed to work.<p>&gt; it doesn’t have to redo the whole thing on every keypress.<p>This is probably what makes the task seem harder than it is. Incremental parsing is nice, but not mandatory. rust-analyzer and most IntelliJ parsers re-parse the whole file on every change (IJ does incremental <i>lexing</i>, which is simple).<p>&gt; The reason (most) LSP servers don’t offer syntax highlighting is because of the drag on performance.<p>I am surprised to hear that. We never had performance problems with highlighting on the server in rust-analyzer. I remember that for Emacs specifically there were client side problems with parsing LSP JSON.<p>&gt; Every keystroke you type must be sent to the server, processed, a partial tree returned, and your syntax highlighting updated.<p>That’s not the bottleneck for syntax highlighting, typechecking is (and it’s typechecking that makes highlighting especially interesting).<p>In general, my perception of what’s going on with proper parsing in the industry is a bit different. I’d say status quo from five years back boils down to people just getting accustomed to the way things were done. Compiler authors generally didn’t think about syntax highlighting or completions, and editors generally didn’t want to do the parsing stuff. JetBrains were the exception, as they just did the thing. In this sense, LSP was a much-needed stimulus to just start doing things properly. People were building rich IDE experiences before LSP just fine (see dart analyzer), it’s just that relatively few languages saw it as an important problem to solve at all.

Before this conversation is railroaded by talk about language servers, as the article points out, tree sitter tends to need to be a bit closer to the environment to be effective.<p>There’s still work to do, but having tree sitter in neovim feels like a great step forward.

&gt; Semantic Bovinator<p>Heh. A long time ago I wrote a video game[1] somewhat similar to Williams Defender, and casting about for some sort of &quot;theme&quot; for the game, I hit upon the &quot;editor wars&quot;, the ancient storied battle between vi and emacs. You are ostensibly &quot;vi&quot;, (a little spaceship vaguely reminiscent of the Vipers from Battlestar Galactica) cruising through system memory, evading system processes, GDB instances, etc trying to recover your &quot;.swp&quot; files. How to represent Emacs? Obviously, via a giant blimp! and I could display all sorts of messages on the side of the blimp, singing the praises of Emacs, and disparaging fans of vi. And the Emacs blimp had a &quot;memory leak&quot;, which meant that pieces of the xemacs source code would literally leak out of the back end of the blimp, with the letters floating lazily away, like smoke. So that meant I had to take a look at the xemacs source, dig through it and try to find some funny bits to put in. Of course, &quot;semantic bovinate&quot; jumped out at me.[2]<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;smcameron&#x2F;wordwarvi" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;smcameron&#x2F;wordwarvi</a> [2] <a href="https:&#x2F;&#x2F;github.com&#x2F;smcameron&#x2F;wordwarvi&#x2F;blob&#x2F;master&#x2F;wordwarvi.c#L1719" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;smcameron&#x2F;wordwarvi&#x2F;blob&#x2F;master&#x2F;wordwarvi...</a>

Checkout the project page here: <a href="https:&#x2F;&#x2F;tree-sitter.github.io&#x2F;tree-sitter&#x2F;" rel="nofollow">https:&#x2F;&#x2F;tree-sitter.github.io&#x2F;tree-sitter&#x2F;</a><p>Quite a lot of languages are already supported, it&#x27;s really nice to see. I might have a use for such a library for a personal project :)<p>You can play around with the playground here: <a href="https:&#x2F;&#x2F;tree-sitter.github.io&#x2F;tree-sitter&#x2F;playground" rel="nofollow">https:&#x2F;&#x2F;tree-sitter.github.io&#x2F;tree-sitter&#x2F;playground</a>

I suppose that these days I am one of the few professional programmers who has an active dislike of syntax highlighting. I find it immensely distracting. The only stuff I allow the highlighter to touch are my comments (I turn them bold) and I consider this a somewhat frivolous indulgence.<p>(I appreciate the complexity of the problem, btw)

I am in love with language servers, the quality of life improvement is just unreal.

I&#x27;m a maintainer of a cli HTTP client with a text plain file format, Hurl [1]. I would like to begin to add support for various IDE (VSCode, IntelliJ), starting from syntax highlighting, but I have hard time to start.<p>I struggle on many &quot;little&quot; details, for instance: syntax error should be exactly the same in the terminal and in the IDE. Should I reimplement exactly the same parsing or should I reuse some of the cli tools parser? If I reuse it, how do I implement things given than, for instance, IntelliJ plugin are written in Java&#x2F;Kotlin, while VScode plugin are Javascript&#x2F;TypeScript, and Hurl is written in Rust...<p>Very hard to figure all when it&#x27;s not your core domain,<p>[1] <a href="https:&#x2F;&#x2F;hurl.dev" rel="nofollow">https:&#x2F;&#x2F;hurl.dev</a>

tree-sitter is a great framework. I have used it quite a bit in past. I even created a small library on top of it, called tree-hugger (<a href="https:&#x2F;&#x2F;github.com&#x2F;autosoft-dev&#x2F;tree-hugger" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;autosoft-dev&#x2F;tree-hugger</a>) Really enjoyed their playground as well.

&gt; The reason (most) LSP servers don’t offer syntax highlighting is because of the drag on performance. Every keystroke you type must be sent to the server, processed, a partial tree returned, and your syntax highlighting updated. Repeat that up to 100 words per minute (or whatever your typing speed is) and you’re looking at a lot of cross-chatter that is just better suited for in-process communication.<p>While I agree... he might be surprised to know that that is what all language servers do anyway, even if they don&#x27;t provide syntax highlighting. Every keystroke gets sent over the LSP. As JSON. It&#x27;s amazing it works as well as it does.

Not coming from the vim&#x2F;eMacs world, I fail to understand what treesitter is compare to a language server? Why would I need both?