TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

pg_duckdb: Splicing Duck and Elephant DNA

186 点作者 jonbaer9 个月前

11 条评论

leetrout9 个月前
If &#x2F; as we move to analytic workloads it would be awesome to see postgres pickup support for AsOf, time_bucket, etc that duckdb and timescale have.<p>I don&#x27;t and never have enjoyed SQL and I much prefer the ergonomics of time_bucket to date_bin.<p>For example, I would do this in duckdb:<p><pre><code> SELECT count(*) as y , time_bucket(interval &#x27;2 weeks&#x27;, at::timestamp) as x FROM analytics WHERE some_bool AND some_haystack = &#x27;needle&#x27; GROUP BY x ORDER BY x </code></pre> In postgres it looks more like:<p><pre><code> with counts as ( SELECT date_bin(&#x27;1 hour&#x27;::interval, at, (now() - interval &#x27;2 weeks&#x27;)::timestamp) , count(*) c FROM analytics a WHERE some_bool AND some_haystack = &#x27;needle&#x27; GROUP BY date_bin ) select series as x, coalesce(counts.c, 0) as y from generate_series( (now() - interval &#x27;2 weeks&#x27;)::timestamp, now()::timestamp, interval &#x27;1 hour&#x27; ) series LEFT JOIN counts ON counts.date_bin = series;</code></pre>
评论 #41280597 未加载
评论 #41283068 未加载
craigkerstiens9 个月前
Very much agreed with this general idea, and believe a lot of this was inspired by the team we hired at Crunchy Data to build it as they were socializing it for a while. Looking forward to pg_duckdb advancing in time for now it still seems pretty early and has some maturing to do. As others have said, it needs to be a bit more stable and production grade. But the opportunity is very much there.<p>We recently submitted our (Crunchy Bridge for Analytics-at most broad level based on same idea) benchmark for clickbench by clickhouse (<a href="https:&#x2F;&#x2F;benchmark.clickhouse.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;benchmark.clickhouse.com&#x2F;</a>) which puts us at #6 overall amongst managed service providers and gives a real viable option for Postgres as an analytics database (at least per clickbench). Also of note there are a number of other Postgres variations such as ParadeDB that are definitely not 1000x slower than Clickhouse or DuckDB.
评论 #41280317 未加载
评论 #41279620 未加载
评论 #41283082 未加载
评论 #41280153 未加载
wenc9 个月前
Would be helpful to list the features. This link has the details:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;duckdb&#x2F;pg_duckdb">https:&#x2F;&#x2F;github.com&#x2F;duckdb&#x2F;pg_duckdb</a><p>Sounds like it would be useful for Postgres users to interact with Parquet and CSV data within a single SQL query and in a performant way (due to DuckDB&#x27;s vectorization).
nikita9 个月前
Postgres IS missing an analytics engine. benchmark.clickhouse.com puts it at the bottom of the list and ~1000x slower than @duckdb and @ClickHouseDB.<p>Here are the scenarios and how to address them<p>1. Query Parquet and Iceberg from Postgres. When Parquet files are stored in S3 Postgres should be able to run analytical queries on them.<p>2. Postgres should allow creation of columnstore tables inside Postgres storage subsystem. Analytical queries on top of these table should be FAST. Top 10 on Clickbench fast. This allows to run analytics without S3 and have super low latencies for analytics.<p>3. Postgres should allow creation of secondary columnstore indexes to speed up analytical queries in mixed workloads. This is super useful for Oracle migrations since Oracle had this feature for a while.<p>So How do we get there? 10 years ago it would be a MASSIVE project, but today we have @duckdb - super fast analytical engine with an open license. The work is still not trivial, but it is much much simpler.<p>First you need to integrate an analytical query processor into Postgres and today @duckdblabs announced github.com&#x2F;duckdb&#x2F;pg_duck…. Yay and congrats!<p>This plugin runs duckdb alongside with Postgres and integrated Postgres syntax with the @duckdb query processor (QP)<p>With that it now can trivially query external files from S3. This addresses scenario 1.<p>With that it now can trivially query external files from S3. This addresses scenario 1.<p>Building columnar table requires either implementing columnar storage from scratch or integrating duckdb storage into the Postgres subsystem. You can of course let duckdb create duckdb files on local disk, but then all the Postgres machinery: replication, backup, recovery won&#x27;t work<p>Duckdb tables have to mapped into 8kb Postgres pages pushed through the Postgres WAL for replication, recovery and transactionality. This will give us scenario 2<p>Scenario 3 is even more work. You need secondary index maintenance and it will require hybrid query execution. We will need to modify Postgres executor so that it can mix and match regular Postgres query operators and &quot;vectorized&quot; query operators from duckdb. Or built vectorized operators into Postgres<p>Scenarios 2 and 3 will take some time, but I&#x27;m excited for this roadmap: this will unlock a huge world for millions of Postgres users and simplify the lives of many developers dealing with moving data between transactional and analytical systems.
评论 #41278692 未加载
评论 #41283108 未加载
评论 #41279894 未加载
saisrirampur9 个月前
Excited about the work here. However, my 2 cents - for this to become a reality (serious production use at scale), I don’t think it’s just based on the choice of the analytical engine (here, DuckDB), but rather on how well the Postgres extension is built. The Postgres extension framework is complex, still maturing, and doesn’t offer full flexibility to implement features. We saw this closely at Citus. It was a deterrent to competing with native analytical databases like ClickHouse and Snowflake. A bunch of customers, including CloudFlare and Heap, switched from Citus to ClickHouse and SingleStore, respectively. This was one of the inspirations to start PeerDB , to make it magical for customers to move data from Postgres to native and purpose-built analytical databases like ClickHouse.<p>Being a Postgres fan, Good luck and best wishes with the effort here!
scirob9 个月前
how are we doing with each of these extensions [ pg_duckdb, pg_analytics (paradedb) , crunchy analytics ] sql syntax being compatible ? (or maybe standardizing on <a href="https:&#x2F;&#x2F;substrait.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;substrait.io&#x2F;</a> ) .<p>It would be great if one could have a diversity of postgres&#x27;s in a data mesh but you can execute the same exact sql on them
评论 #41284831 未加载
timenova9 个月前
Looking forward to this getting supported on Neon!
评论 #41279287 未加载
grepLeigh9 个月前
Does the pg_duckdb extension double-write data to DuckDB? I skimmed the blog post and didn&#x27;t understand if the extension adds some capabilities to Postgres, or just makes it easier to maintain a consistent data model across two database engines.
StimDeck9 个月前
They had to use duck DNA and not pig DNA because, as everyone already knows, pig and elephant DNA just don’t splice.
coatue9 个月前
repo: <a href="https:&#x2F;&#x2F;github.com&#x2F;duckdb&#x2F;pg_duckdb">https:&#x2F;&#x2F;github.com&#x2F;duckdb&#x2F;pg_duckdb</a>
netcraft9 个月前
this looks awesome, I cant wait to play with it! we need to get this into RDS as soon as we can