ANTLR Mega Tutorial

I last played around with ANTLR in 2012 (when it was version 3) and I discovered that there&#x27;s a &quot;bigger picture&quot; to the parser generator universe that most tutorials don&#x27;t talk about:<p>1) ANTLR is a good tool for generating &quot;happy path&quot; parsers. With a grammar specification, it easily generates a parser that accepts or rejects a piece of source code. However, it&#x27;s not easy to use the hooks to generate high quality diagnostic error messages.<p>2) ANTLR was not good for speculative parsing or probabilistic parsing which would be the basis of today&#x27;s generation of tools such as &quot;Intellisense&quot; not giving up on parsing when there&#x27;s an unclosed brace or missing variable declaration.<p>The common theme to the 2 bullet points above is that a high quality compiler written by hand will hold multiple &quot;states&quot; of information and an ANTLR grammar file doesn&#x27;t really have an obvious way to express that knowledge. A pathological example would be the numerous &quot;broken HTML&quot; pages being successfully parsed by browsers. It would be very hard to replicate how Chrome&#x2F;Firefox&#x2F;Safari&#x2F;IE doesn&#x27;t choke on broken HTML by using ANTLR to generate an HTML parser.<p>In short, ANTLR is great for <i>prototyping</i> a parser but any industrial-grade parser released into the wild with programmers&#x27; expectations of helpful error messages would require a hand-written parser.<p>Lastly, the lexing (creating the tokens) and parsing (creating the AST) is a <i>very tiny percentage</i> of the total development of a quality compiler. Therefore, ANTLR doesn&#x27;t save as much time as one might think.<p>I welcome any comments about v4 that makes those findings obsolete.

I switched over to ANTLR 4. It is strictly superior to ANTLR 3. The listener approach rather than embedding code in the grammar is very natural. Separating leads to clean grammars and clean action code. Odd thing is that I was stuck on 3 because 4 didn&#x27;t support C yet and then I just switched to the Java target in an anti-C pique. Shoulda done that awhile ago.<p>TParr&#x27;s <i>The Definitive ANTLR 4 Reference</i> is quite good. And so&#x27;s this mega tutorial.<p><a href="https:&#x2F;&#x2F;pragprog.com&#x2F;book&#x2F;tpantlr2&#x2F;the-definitive-antlr-4-reference" rel="nofollow">https:&#x2F;&#x2F;pragprog.com&#x2F;book&#x2F;tpantlr2&#x2F;the-definitive-antlr-4-re...</a><p>ANTLR is my goto tool for DSLs.

I think everyone should manually implement a simple recursive descent parser at least once in their careers. It&#x27;s surprisingly easy, and really (in my experience) helps to break through the mental barrier of parsers being magical black boxes.<p>Plus, once you have an understanding of recursive descent parsing, it&#x27;s a relatively small leap to recursive descent code generation. And once you&#x27;re there, you have a pretty good high-level understanding of the entire compilation pipeline (minus optimization).<p>Then all of a sudden, compilers are a whole lot less impenetrable.

Just a note on &quot;Why not to use regular expression&quot;. Because it&#x27;s <i>impossible</i> depending on the language complexity<p>REs are level 3 <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Chomsky_hierarchy" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Chomsky_hierarchy</a>

I used ANTLR to write a fuzz testing tool which parses an ABNF grammar (like in an IETF RFC) and then generates random output which matches the grammar. Worked great!<p><a href="https:&#x2F;&#x2F;github.com&#x2F;nradov&#x2F;abnffuzzer" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;nradov&#x2F;abnffuzzer</a>

Great tutorial, ANTLR is one of the best tools for prototyping languages and compilers.<p>I wasn&#x27;t aware it supports JavaScript nowadays.<p>In any case, good selection of languages.

For .Net projects, I&#x27;ve used Irony. From the CodePlex site:<p>&quot;Unlike most existing yacc&#x2F;lex-style solutions Irony does not employ any scanner or parser code generation from grammar specifications written in a specialized meta-language. In Irony the target language grammar is coded directly in c# using operator overloading to express grammar constructs. &quot;

I used Antlr v3 to create a NLP parser for calendar events for iOS and Android. It took longer than expected. iOS + C runtime was opaque, so had to write a tool for debugging. Android + Java runtime overran memory, so had to break into separate grammars. Of course, NLP is not a natural fit. Don&#x27;t know what problems are fixed by v4.<p>&gt; The most obvious is the lack of recursion: you can’t find a (regular) expression inside another one ...<p>PCRE has some recursion. Here is an example for parsing anything between { }, with counting of inner brackets:<p>&#x27;(?&gt;\{(?:[^{}]<i>|(?R))</i>\})|\w+&#x27;<p>A C++11 constexpr can make hand coded parsers a lot more readible, allowing token names in case statements. For example , search on &quot;str2int&quot; in the following island parser: <a href="https:&#x2F;&#x2F;github.com&#x2F;musesum&#x2F;par" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;musesum&#x2F;par</a>

These types of tutorials always start out explaining the problems with Regular Expressions and why not to use them... then immediately proceeding into lexing via regular expressions.<p>Perhaps the tutorials should start with the strengths of regular expressions, and how we can harness that for getting started with a lexer.

What are the advantages of ANTLR over something like Haskells Parsec?<p><a href="https:&#x2F;&#x2F;github.com&#x2F;aslatter&#x2F;parsec" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;aslatter&#x2F;parsec</a>

This tutorial looks great. I picked up Antlr4 a few months ago, and hadn&#x27;t done any parsing before then. The first week was basically me, The Definitive Antlr4 Reference, and extreme confusion with how different targets worked. Compounding the problem was the fact that a lot of the antlr4 example grammars only work for a specific target. The use of different language implementations as part of this tutorial seems really useful!<p>(Antlr4 is awesome :)

I used ANTLR to write a Python parser for the SNOMED expression language last year, and testing it was one of the weirder parts of the experience. I was up and running in a few days, which was largely thanks to the ANTLR book. I love this project. It made doing what I did a lot more fun than I thought it would be. Hand-rolling an ABNF parser from scratch would be a nice hobby project, but not when one has a deadline.