Citus 10 brings columnar compression to Postgres

Beware that simply adding a column-oriented storage engine to a row store like Postgres is not going to get you anywhere near the performance of a ground-up columnar system like Redshift or Snowflake. This paper explains why [1]. Short version: most of the benefits are in the column-oriented execution engine, which differs in every aspect of its implementation from a row-oriented execution engine.<p>[1] <a href="https:&#x2F;&#x2F;stratos.seas.harvard.edu&#x2F;files&#x2F;stratos&#x2F;files&#x2F;columnstoresfntdbs.pdf" rel="nofollow">https:&#x2F;&#x2F;stratos.seas.harvard.edu&#x2F;files&#x2F;stratos&#x2F;files&#x2F;columns...</a>

One of the gotchas of columnar storage (coming from Redshift) is that you lose all of the compression benefits if you have just one column that’s fat or hard to compress.<p>In Redshift columns are stored in blocks. You want to fit roughly the same number of column values per block across all your columns. But if you have one column where a small number of values can fit in a block, the rest of the columns end up leaving most of the block space unused. The result is wasted disk space and poor query performance.<p>This Postgres extension has similar-sounding storage ideas with stripes, but it’s not clear to me if it suffers from the same issue.<p>My first test to vet this would be a table with 50 columns of ints and one column of md5 hashes stored as varchar.

I&#x27;m curious to see how this compares in real life to TimescaleDB hypertables with compression - which to me, reads as much the same thing. I&#x27;m wondering if Citus is bringing a lower level implementation of idea possibly?

Reassuring to see big new features like this coming out after the Microsoft acquisition, a healthy sign that this open source product continues to see serious ongoing investment.

Great start! Keep in mind the limitations:<p>What are the Limitations?<p>These limitations are not set in stone, and we look forward to working on them in the future:<p>No UPDATE or DELETE support No index support No logical replication or logical decoding support See more limitations in the columnar README

Amazon really missed the boat on Citus, a few more great acquisitions like this and Azure is going to look great (yes I know it can be still self installed on AWS).

I am glad to see Postgresql ecosystem growing. It gets better by the day. Thanks to all people involved!

since we&#x27;re talking postgresql:<p>i recently started diving into postgresql and it seems to me that there is a patchwork of HA solutions (with patroni being the most feature-full) but no real multi-master solution for postgresql released under an open source license.<p>There&#x27;s BDR (bi-directional replication) but apparently 2ndquadrant pulled it back under a proprietary license, am i right?<p>what&#x27;s the current status of postgresql multi-master HA?

Awesome news! After seeing the benefits with the cstore extension, I assumed this was only a matter of time.

So are they using Parquet, ORC or Arrow under the hood; or do they have a custom format?

Memory, IO, Computation and DB connection concurrency all 4 have to be kept in the mind to achieve optimum latency across wide range of complex queries.<p>It has been my observation that if you follow 3NF design, table partition and carefully crafted Materialized Views works 8 out of 10 times when data size is huge.

Sensage&#x2F;Addamark was too early to the columnar storage game in 2001-2003 ... <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Sensage" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Sensage</a>.