Buckets of Parquet Files Are Awful

Everything in this article implies a lack of understanding of the more common use cases for Parquet - data sets that exceed typical single node RAM sizes. The article would make more sense titles &quot;Large Buckets of Parquet Files Are Awful on a Laptop&quot;.<p>Don&#x27;t read Parquet into memory in one pass. Use an engine that can scan it, ex. Spark.<p>PostgreSQL and Clickhouse have their own storage engines, and bulk importing _large_ volumes of data no matter the format (Parquet, CSV, JSON) will present challenges.<p>DuckDB is meant to be a single node engine. If your data size exceeds RAM, its going to be a problem.<p>If your data is not partitioned appropriately for its size, that is a problem with the people who created it.

buckets of shard files are awesome. many massive pipelines doing important work use that model. it&#x27;s simple and easy to program around. Inspecting data and sampling it is straightforward. Even record files (with no index) can be easily seeked and sampled. They are extremely efficient by reducing the number of inodes. Tools like LexicographicRangeSharding can be used to make the bucket object sizes more balanced.<p>Use sampling to make the data size more reasonable for doing local work. But, every tool I&#x27;ve used to deal with large data has handled large data the way I expect: either loading whole files into RAM, mapping whole files into RAM, or using fixed buffers. Typically when my work exceeds RAM, I just move to a machine with bigger RAM but still work with large shard files.

I would say on the contrary, please stop dumping CSV data just because that’s all you know all to deal with.<p>CSV is full of gotchas which is why CSV readers have so many switches.<p>They are huge (and hugely inefficient at scale), have no type safety, don’t support predicate push down and are only meant to be read linearly.<p>Most of the OPs complaints are about loading and memory, not about actually using the data. OP is more concerned about infra and memory usage than actually the end goal: using the the data well.<p>I’m not saying Parquet is the be all and end all of data formats but it’s a darned sight better than CSVs for almost all analytics use cases.<p>People who use CSV because it’s readable and understandable either have non analytics use cases or just don’t want to learn anything new.

Ok, I guess I am repeating myself but so does this post since I assume it spawned from this discussion <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40284225">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40284225</a><p>Why does no-one mention Trino or the AWS branded Trino - Athena?<p>This will solve basically all of the problems listed.<p>In the case of Athena and provided you already have your data in s3, without having to set up a new service and very little work.<p>Using Athena is not like setting up a new database. You just point Athena to the files and it will use resources provided by AWS to query. No import required, no service required.<p>I think Athena is an amazing tool. I can borrow like 120 CPUs (my tasks often parallelize to that) for free during 5 minutes or even an hour to join &#x2F; group &#x2F; modify all my data.

I just built an NVMe cache when a customer of mine had the problem of reading parquet from S3. Basically first hit goes through the ram, but then it also writes it to NVMe. The next hit just checks etag and if it hasn’t changed, mmaps the file directly from the cached version for faster access. Eviction is typical LRU, but with leases. When disk cache is full (and sufficient storage can’t be released because files are in use) it only uses DRAM. When DRAM is full, which is very unlikely, the cache blocks and waits until stuff gets released.<p>I’d think twice before abandoning a more or less standard, open format for some brand new shiny thing.

I&#x27;ve recently played with the Overture Maps datasets which are in .parquet and I agree that there is a learning curve.<p>One thing that&#x27;s nice about .parquet is typical Macs come with 128&#x2F;256GB SSDs and you can query 300GB of parquet in S3 without needing all those files on your local hard drive. Some of these queries are also surprisingly fast but I haven&#x27;t looked into how it works.<p>Edit: Having used MySQL and Postgres, I actually think DuckDB is better than either.

I don’t think buckets of csv would be better, in fact I know it would be worse because you lose data types and in memory representation.

Postgres CAN read from parquet using parquet_fdw, and then you can import directly using &quot;create table x as select&quot; from the parquet foreign tables.<p>&gt;Postgres can&#x27;t read from S3, load from Parquet, or query parquet files.

A big part of my job is to ingest data from hundreds of providers to process.<p>A provider that will push data to my S3 bucket, as parquet files, is the best I can think of.<p>I don&#x27;t want no provider managed database to host, stupidly huge zipped CSV dumps, or whatever weird technology the provider can think of.<p>Seriously reading a file from S3 and stream processing the parquet is not the peak of engineering. What is there to complain about...<p>Some of the insanity I had to deal with:<p>- 350TB of zipped CSVs which columns and delimiter vary across time, shipped on HDD by mail<p>- Self hosting a MSSQL server for the provider to update<p>- Good old FTP polling at more or less predictable times<p>- Fancy you-name-it-shiny-new-tech remote database accounts<p>- The crappy client http website with custom SQL inspired query language to download zipped extracts<p>- The web 2.0 rest API, billed per request<p>- Snowflake data warehouse data lake big data<p>Please, don&#x27;t be that guy, just push parquet flat files in my bucket.

The arguments presented by the author reminds me of the saying &quot;Incompetent dancer complained that the stage was crooked&quot;

This article would be greatly improved by at least a single sentence saying what your proposed alternative is. Otherwise it sounds like you&#x27;re saying &quot;having to eat is really annoying. let&#x27;s stop eating&quot;.

It&#x27;s quite clear that this blog post is in direct response to this comment: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40299364">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=40299364</a><p>Nothing wrong with that, but good for folks to know the context.

Things become popular because they are good&#x2F;decent&#x2F;better than the alternatives.<p>Then, popular things become the default choice even when they don&#x27;t necessarily make sense because they are not solving the problems they originally meant to solve<p>Then people complain about it

This is the one thing that Azure does better than AWS. With Azure Synapse, you can upload parquet files to a storage account (similar to an AWS bucket), create a view in a serverless SQL instance, and interact with it via any SQL Server client.

They are not awful if you use right tools. Welcome to the world of Spark, Apache Drill, Trino&#x2F;Presto, etc.