Parquet: An efficient, binary file format for table data

There is a point which one needs to be aware of with Parquet: if you are not stepping into it with tools well suited for it (that is: Go, C++, Java or Python codebase/runtime, mainstream platform...) you are going to have a bad time. For all its advantages, Parquet is _very_ complicated (IMO overcomplicated, on multiple fronts), and uses binary dependencies which you absolutely need to carry around (Thrift?..). Up to a certain data size the advantage of CSV is not that it is a better format (it is decidedly, absolutely worse/more ambiguous than Parquet) but the fact that you can write a valid CSV in a couple of lines of anything, and read in a couple of lines of anything. If you ask me, Parquet - for all its advantages - is a product of the Cambrian explosion of the Apache-big-data-projects. Very little in its design warrants (or requires) using things like Thrift, so I won't exclude that something less overcomplicated (and more welcoming, and requiring less dependencies) will appear on the scene at some point.

This is all very interesting (and makes me check out Parquet) but it's painful to see how the text goes to great lengths to describe how it avoids common problems and what they are, but doesn't lose a single word on... how they actually solved these problems. What is the actual boolean type? Which encoding are you actually working in? There's nothing concrete but an implication that it's also compressing data?

“Parquet has both a date type and the datetime type (both sensibly recorded as integers in UTC).”<p>What does it mean for a date to be utc? my date, but in the utc timezone? Usually when I write a date to a file, I want a naive date; since that’s the domain of the data. 2020-12-12 sales: $500. But splitting that by some other timezone seems to be introducing a mistake.<p>Often I want to think in local naive time too, especially for things like appointments that might change depending on dst or whatever. Converting to utc involves some scary things. Timestamps are also useful but I don’t want to transcode my data sometimes as the natural format is the most correct.

One of the main interest of csv is that it&#x27;s human readable and usable with common text tools (such as grep, cat or any texte editor) so I&#x27;m not sure the comparison should be done between parquet and csv, but between parquet and other formats like hdf5 and netcdf. What is the advantage of this new format compared to the older ones ?

And if you are interested on encoding Geospatial Data there is a format called Geoparquet that is on the way of standarization: <a href="https:&#x2F;&#x2F;geoparquet.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;geoparquet.org&#x2F;</a> Essentially adding metadata on the extensible schema metadata streucture that parquet supports. Think of WKB in a column. With more exciting stuff coming that way.

Oh very interesting didn’t expect so much scepticism on parquet. We use it to store around 100 tb of data in a partitioned way on GCS and query it using BigQuery. Absolutely fantastic solution and incredibly cheap too.

Can you `grep` it?<p>Text-based files like CSV can be `grep`-ed en masse, which I do often.<p>eg, to find some value across a ton of files.<p>Is that possible with parquet?

S3 Select supports parquet, which means you can push query to individual S3 blobs. I think there is a possibility we might see someone build a postgres foreign data wrapper which pushes the querying via S3 select to S3 blobs and use postgres only for the final aggregations. Someone on HN had done this with sqlite files in S3 queried by lambdas using sqlite, but pushing it wholly to S3 would eliminate a lot of overhead.

Parquet fits well into a system like Apache Arrow for good query performance, when considering partitioning (via directories), push-down predicates, and read-only-the-columns-you-need: <a href="https:&#x2F;&#x2F;arrow.apache.org&#x2F;docs&#x2F;python&#x2F;dataset.html" rel="nofollow">https:&#x2F;&#x2F;arrow.apache.org&#x2F;docs&#x2F;python&#x2F;dataset.html</a> &quot;Tabular Datasets&quot;

I was surprised lately to realize there is a not a standard way to import a parquet file in MSSQL. Happy to be proven wrong

If you need a quick tool to convert your CSV files, you can use csv2parquet from <a href="https:&#x2F;&#x2F;github.com&#x2F;domoritz&#x2F;arrow-tools">https:&#x2F;&#x2F;github.com&#x2F;domoritz&#x2F;arrow-tools</a>.

I&#x27;ve used Parquet here to filter&#x2F;map 400 Million Twitter geotagged tweets [1]. The main advantage was that it automatically partitions data so it can be streamed and chunked during processing.<p>[1]: <a href="https:&#x2F;&#x2F;ad.vgiscience.org&#x2F;twitter-global-preview&#x2F;00_Twitter_datashader.html" rel="nofollow">https:&#x2F;&#x2F;ad.vgiscience.org&#x2F;twitter-global-preview&#x2F;00_Twitter_...</a>

&gt; On a real world 10 million row financial data table I just tested with pandas I found that Parquet is about 7.5 times quicker to read than csv, ~10 times quicker to write and a about a fifth of the size on disk. So way to think of Parquet is as &quot;turbo csv&quot; - like csv, just faster (and smaller).<p>Come on... who does tests like this? Why is anyone supposed to believe these numbers? What is even the point of making such claims w&#x2F;o <i>any</i> evidence?<p>---<p>PS. Also, buried somewhere in the middle of the article is the admission that the format isn&#x27;t possible to stream.<p>And, if we expand on the file structure, then it becomes apparent that it was inspired by typical relational database storage, except simplified, which made the format awful for insertion &#x2F; deletion operations. Of course, if you compare to CSV, then there&#x27;s no loss here, but there are other formats which can handle limited, or even unlimited insertion &#x2F; deletion with the resource use similar to writes.<p>Even truncation in this format is complicated (compared to CSV).<p>Seems like the format was trying to be the middle ground between proper relational database storage and something like CSV, but not sure if such middle ground is actually necessary...

Parquet is cool. What isn&#x27;t so cool is that, e g., partitioning isn&#x27;t supported on all platforms out of the box. Hence, reading a set of files created in one language may require extra work in another language. Partitioning should be supported in all languages.

I wanted to dive into Parquet internals a couple of times to see how it works but there seems to be no tool to see page level data. The only option seems to be to use Arrow’s C++ library. Is anybody aware of a higher level option?

Why not use Delta? <a href="https:&#x2F;&#x2F;www.vldb.org&#x2F;pvldb&#x2F;vol13&#x2F;p3411-armbrust.pdf" rel="nofollow">https:&#x2F;&#x2F;www.vldb.org&#x2F;pvldb&#x2F;vol13&#x2F;p3411-armbrust.pdf</a>

i wonder why the words &#x27;apache&#x27;, &#x27;twitter&#x27;, &#x27;cloudera&#x27;, &#x27;cutting&#x27;, &#x27;rcfile&#x27;, &#x27;orc&#x27;, &#x27;cutting&#x27;, &#x27;trevni&#x27;, &#x27;thrift&#x27;, &#x27;hadoop&#x27;, &#x27;julien&#x27;, &#x27;jco&#x27;, &#x27;tianshuo&#x27;, and &#x27;deng&#x27; don&#x27;t appear in this page<p>i mean i understand not mentioning <i>all</i> of them but it&#x27;s pretty disappointing to not mention <i>any</i> of them

I&#x27;m interested in a good tool for converting csv to pq. Seems many encoding features are not supported by most.

I&#x27;ve still yet to figure out where feather fits into this.

Why is the picture of Sutjeska war memorial in the article?

<p><pre><code> apt-cache search parquet &lt;nada&gt;</code></pre>

Why should not use a simple database instead ?