Author here. Since I wrote this, Arrow seems to be be more and more pervasive. As a data engineer, the adoption of Arrow (and parquet) as a data exchange format has so much value. It's amazing how much time me and colleagues have spent on data type issues that have arisen from the wide range of data tooling (R, Pandas, Excel etc. etc.). So much so that I try to stick to parquet, using SQL where possible to easily preserve data types (pandas is a particularly bad offender for managing data types).<p>In doing so, I'm implicitly using Arrow - e.g. with Duckdb, AWS Athena and so on. The list of tools using Arrow is long! <a href="https://arrow.apache.org/powered_by/" rel="nofollow">https://arrow.apache.org/powered_by/</a><p>Another interesting development since I wrote this is DuckDB.<p>DuckDB offers a compute engine with great performance against parquet files and other formats. Probably similar performance to Arrow. It's interesting they opted to write their own compute engine rather than use Arrow's - but I believe this is partly because Arrow was immature when they were starting out. I mention it because, as far as I know, there's not yet an easy SQL interface to Arrow from Python.<p>Nonetheless, DuckDB are still Arrow for some of its other features: <a href="https://duckdb.org/2021/12/03/duck-arrow.html" rel="nofollow">https://duckdb.org/2021/12/03/duck-arrow.html</a><p>Arrow also has a SQL query engine: <a href="https://arrow.apache.org/blog/2019/02/04/datafusion-donation/" rel="nofollow">https://arrow.apache.org/blog/2019/02/04/datafusion-donation...</a><p>I might be wrong about this - but in my experience, it feels like there's more consensus around the Arrow format, as opposed to the compute side.<p>Going forward, I see parquet continuing on its path to becoming a de facto standard for storing and sharing bulk data. I'm particularly excited about new tools that allow you to process it in the browser. I've written more about this just yesterday: <a href="https://www.robinlinacre.com/parquet_api/" rel="nofollow">https://www.robinlinacre.com/parquet_api/</a>, discussion: <a href="https://news.ycombinator.com/item?id=34310695" rel="nofollow">https://news.ycombinator.com/item?id=34310695</a>.