I am very interested in this project as a "better awk" is something I have often fantasized about.<p>I read all of the documentation that's available on <a href="https://tokay.dev/tokay-docs/" rel="nofollow">https://tokay.dev/tokay-docs/</a>, but unfortunately it never really... describes itself? Many sections, including the section on "parselets" are just unwritten. "Consumable" values are mentioned but never described (there is a "stub" section that doesn't really explain what the term means).<p>It begins with a pretty detailed description of value "severity" but doesn't really motivate why the concept exists. (I <i>think</i> that it's (basically) a way to very concisely discard certain matches? When there are "more important" matches around them?)<p>There are no examples of how I could use Tokay to "parse" something -- there are lots of examples dotted through the docs, but none of them demonstrate working with structured file formats, and they feel a little bit contrived.<p>I'm not complaining here: this project is not making any false claims about its status, the docs are clearly and explicitly unfinished, it is very clear that Tokay is still under active development.<p>But I want to learn more about it! I came away from this with a sense that it has the potential to be really useful to me, but without any concrete evidence to support that. I guess the next step is to download the source and start reading through the tests.<p>All this to say: please highlight some examples showcasing situations where Tokay shines! (Parsing CSVs containing quoted strings was making the rounds recently, right? What does that look like in Tokay?)<p>Oh, actually, the GitHub readme has an example that is more involved than any in the documentation: <a href="https://github.com/tokay-lang/tokay" rel="nofollow">https://github.com/tokay-lang/tokay</a><p><pre><code> _ : [ \t]+ # redefine whitespace to just tab and space
Factor : @{
Integer _ # built-in 64-bit signed integer token
'(' _ Expr ')' _
}
Term : @{
Term '*' _ Factor $1 * $4
Term '/' _ Factor $1 / $4
Factor
}
Expr : @{
Expr '+' _ Term $1 + $4
Expr '-' _ Term $1 - $4
Term
}
Expr _ print("= " + $1) # gives some neat result output</code></pre>