SPyQL looks very promising, great work!<p>I can't help but mention clickhouse-local tool: <a href="https://clickhouse.com/docs/en/operations/utilities/clickhouse-local" rel="nofollow">https://clickhouse.com/docs/en/operations/utilities/clickhou...</a><p>clickhouse-local is a single binary that enables you to perform fast data processing using SQL - effectively database features without a database. This tool supports the full breadth of ClickHouse functions, many popular file formats and recently added automatic schema inference. You can query not only local files, but also remote files (from S3/HDFS/static files accessed by URL). Moreover, clickhouse-local tool has interactive mode where you can create tables, play with data and do almost everything that you can do wih ordinary database. And let's not forget, this tool is written in C++, so it's incredibly fast.<p>Disclaimer: Work at ClickHouse
SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions!<p>As far as similar tools go, if you're interested, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2].<p>DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats.<p>dsq is based on SQLite which means it has to load data into SQLite first, but then gives you the whole breath of SQLite, it also supports many data formats, but is slower at the same time.<p>OctoSQL is faster, extensible through plugins, and supports incremental query execution, so you can i.e. calculate and display a running group by + count while tailing a log file. It also supports normal databases, not just file formats, so you can i.e. join with a Postgres table.<p>[0]: <a href="https://github.com/apache/arrow-datafusion" rel="nofollow">https://github.com/apache/arrow-datafusion</a><p>[1]: <a href="https://github.com/multiprocessio/dsq" rel="nofollow">https://github.com/multiprocessio/dsq</a><p>[2]: <a href="https://github.com/cube2222/octosql" rel="nofollow">https://github.com/cube2222/octosql</a><p>Disclaimer: Author of OctoSQL
See also Jeroen Janssens' Data Science at the Command Line:<p><a href="https://datascienceatthecommandline.com/2e/" rel="nofollow">https://datascienceatthecommandline.com/2e/</a>
SPyQL looks fantastic!<p>The thing that worried me when looking into SQL-tools for CSV-files on the commandline, is the plethora of tools available, and it being hard to find one that feels solid and well-supported enough to become a "default" tool for many daily tasks.<p>I want to avoid investing a lot of time learning the ins and outs of a tool that might stop being developed in a year from now. I wish for something that can become the "awk of tomorrow", but based on SQL or something similar.<p>Does anyone have any experiences related to that? Is my worry warranted? Are some projects more well supported than others?
Once your data is at a certain size, it might be worth considering tools that does the job quickly enough while still being simple to use. This comparison is very interesting:<p><a href="https://colab.research.google.com/github/dcmoura/spyql/blob/master/notebooks/json_benchmark.ipynb" rel="nofollow">https://colab.research.google.com/github/dcmoura/spyql/blob/...</a><p>Disclaimer: Work at ClickHouse, whose tool is part of the benchmarking efforts linked to above.
The best part is that doing analytics via the command line often means that you're doing analytics locally, which often gets you performance superior to a small computing cluster.
Very useful, seems to be an effective bridging tool between relational and NoSQL database types, and from the command line! Nice clear documentation page as well.
Was I the only one thinking of something like google analytics but for command line? A system of usability telemetry for command line utilities might be useful?