I wanted so much to use csvkit and all the features it had, but its horrendous performance made it unscalable and therefore the more I used it, the more technical debt I accumulated.<p>This was one of the reasons I wrote zsv (<a href="https://github.com/liquidaty/zsv">https://github.com/liquidaty/zsv</a>). Maybe csvkit could incorporate the zsv engine and we could get the best of both worlds?<p>Examples (using majestic million csv):<p>---<p>csvcut -c 1,3 = 5.3 seconds<p>zsv select -n -- 1 3 = 0.19 seconds<p>28x faster<p>---<p>csvsql --query "select count(*) from file" file.csv = 148 seconds<p>zsv sql "select count(*) from data" file.csv = 0.68 seconds<p>216x faster<p>---
It is interesting how much different tools vary in their performance for the same task. For example R with data.table is <i>much</i> faster than base R. And Excel Power Query performance is, well, see for yourself:<p><a href="https://www.easydatatransform.com/data_wrangling_etl_tools.html" rel="nofollow">https://www.easydatatransform.com/data_wrangling_etl_tools.h...</a>