This is awesome, thanks for creating this. I've had to write some absolutely wonky scripts to dump a PostgreSQL database into Parquet, or read a Parquet file into PostgreSQL. Normally some terrible combination of psycopg and pyarrow, which worked, but it was ad-hoc and slightly different every time.<p>A lot of other commenters are talking about `pg_duckdb` which maybe also could've solved my problem, but this looks quite simple and clean.<p>I hope for some kind of near-term future where there's <i>some</i> standardish analytics-friendly data archival format. I think Parquet is the closest thing we have now.
Parquet itself is actually not that interesting. It should be able to read (and even write) Iceberg tables.<p>Also, how does it compare to pg_duckdb (which adds DuckDB execution to Postgres including reading parquet and Iceberg), or duck_fdw (which wraps a DuckDB database, which can be in memory and only pass-through Iceberg/Parquet tables)?
Cool, would this be better than using a clickhouse / duckdb extension that reads postgres and saves to Parquet?<p>What would be recommended to output regularly old data to S3 as parquet file? To use a cron job which launches a second Postgres process connecting to the database and extracting the data, or using the regular database instance? doesn't that slow down the instance too much?
<a href="https://github.com/pgspider/parquet_s3_fdw">https://github.com/pgspider/parquet_s3_fdw</a> is the foreign data wrapper alternative
Why not just federate Postgres and parquet files? That way the query planner can push down as much of the query and reduce how much data has to move about?