Multi-database support in DuckDB

Very cool. I wonder if the pluggable storage engine also supports query pushdown.<p>For example, if you do SELECT ... FROM mytable WHERE key = 42, then if this is a Postgres table, you want the key predicate to be pushed down to Postgres so you don&#x27;t need to scan the whole table. Same for ORDER BY, joins, LIMIT, OFFSET. For joins between foreign tables from different sources, you&#x27;d want the optimizer to be aware of table&#x2F;column stats in order to pick the right join strategy.<p>Sqlite has a &quot;virtual table&quot; mechanism where you can plug in your own engine. But it&#x27;s relatively simplistic when it comes to pushdown.

How big is the DuckDB team? I am blown away at the quality of life improvements they keep bringing into the project.

Exactly what I want from DuckDB. Was playing with using it as a quicker cache for a data app backed in Snowflake, wonder how hard it’d be to write the attach for that vs doing it at the client level

i found this feature by accident while comparing performance of duckdb and clickhouse for materializing from an sqlite file, duckdb was instant vs clickhouse taking seconds to stream the data from sqlite. then i found out duckdb actually uses sqlite directly including the indices that were stored. once the table was in a clickhouse table, clickhouse was twice as fast without an index than duckdb or sqlite with index but the initial load from sqlite took way longer. for one shot things like in a dbt pipeline duckdb might make more sense if all input formats are supported.

Does duckdb support remote-duckdb as a storage engine? Seems like a way to support distributed duckdbs. Ducks all the way down? :)

I wish if there&#x27;s a way we can hook DuckDB as Extension to Postgres itself. Currently there&#x27;s no proper columnar database&#x2F;extension exist in PG. In most of the use cases, one doesn&#x27;t really require dedicated &quot;analytics&quot; only DB and sometimes everything has to be done through &quot;Primary DB&quot; (typically in enterprise products world).<p>By hooking DuckDB as Extension, we get best of both of worlds from PG and DuckDB while retaining the persistence and analytical capabilities without hosting yet another DB infrastructure.

Fantastic!<p>If Kuzu and LanceDB are added we could have a perfect way of dealing with graphs, vector, and tables all together in one ecosystem. Honestly may be worthy of a thin wrapper project around those.<p>They would work so incredibly well together to free people from the horrible corporate &quot;cloud&quot; mess&#x2F;B&#x27;s everyone keeps trying to push.

I was just reading about this and wanted to ask: how much do the different storage engines affect performance? I assumed you couldn&#x27;t just plop the DuckDB query engine on top of the PostgreSQL storage engine and get DuckDB performance.

Harsh reality is that in real world scenario different databases would be in different network segment probably not open to each other. Switching from one db to another will limit the capabilities.

What are the performance implications, if any? I assume it&#x27;s just translating queries as needed into the database engine language, in which case I wouldn&#x27;t expect much of a performance issue.

Can someone provide an example of where someone is using some other tech, but when you get to X records, or Y columns, and Z sorts of queries it makes sense to switch from MySQL&#x2F;SQLite&#x2F;PostGres&#x2F;etc to DuckDB?

Can someone explain to me what&#x27;s the upside of having multiple databases? I mean, I understand the downside, no explanation needed there.

What are the limits on the number of databases (say sqlite files) one can have attached (and queried through a UNION) at a single time?

I guess this should allow to connect duckdb to a BigQuery instance since there is a Postgres compatibility layer?

I also wait for their spark emulation too.