We're developing TQL (Tenzir Query Language, "tea-quel") that is very similar to PQL: <a href="https://docs.tenzir.com/pipelines" rel="nofollow">https://docs.tenzir.com/pipelines</a><p>Also a pipeline language, PRQL-inspired, but differing in that (i) TQL supports multiple data types between operators, both unstructured blocks of bytes and structured data frames as Arrow record batches, (ii) TQL is multi-schema, i.e., a single pipeline can have different "tables", as if you're processing semi-structured JSON, and (iii) TQL has support for batch and stream processing, with a light-weight indexed storage layer on top of Parquet/Feather files for historical workloads and a streaming executor.
We're in the middle of getting TQL v2 [@] out of the door with support for expressions and more advanced control flow, e.g., match-case statements. There's a blog post [#] about the core design of the engine as well.<p>While it's a general-purpose ETL tool, we're targeting primary operational security use case where people today use Splunk, Sentinel/ADX, Elastic, etc. So some operators are very security'ish, like Sigma, YARA, or Velociraptor.<p>Comparison:<p><pre><code> users
| where eventTime > minus(now(), toIntervalDay(1))
| project user_id, user_email
</code></pre>
vs TQL:<p><pre><code> export
where eventTime > now() - 1d
select user_id, user_email
</code></pre>
[@] <a href="https://github.com/tenzir/tenzir/blob/64ef997d736e9416e859bfcd5f6fa74970565204/rfc/004-query-language/README.md">https://github.com/tenzir/tenzir/blob/64ef997d736e9416e859bf...</a><p>[#] <a href="https://docs.tenzir.com/blog/five-design-principles-for-building-a-data-pipeline-engine" rel="nofollow">https://docs.tenzir.com/blog/five-design-principles-for-buil...</a>