DeepSeek's smallpond: Bringing Distributed Computing to DuckDB

Deepseek is the real &quot;open&lt;something&gt;&quot; that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing:<p>1. smallpond: <a href="https:&#x2F;&#x2F;github.com&#x2F;deepseek-ai&#x2F;smallpond">https:&#x2F;&#x2F;github.com&#x2F;deepseek-ai&#x2F;smallpond</a><p>2. 3fs: <a href="https:&#x2F;&#x2F;github.com&#x2F;deepseek-ai&#x2F;3FS">https:&#x2F;&#x2F;github.com&#x2F;deepseek-ai&#x2F;3FS</a><p>3. deepep: <a href="https:&#x2F;&#x2F;github.com&#x2F;deepseek-ai&#x2F;DeepEP">https:&#x2F;&#x2F;github.com&#x2F;deepseek-ai&#x2F;DeepEP</a>

It was already on HN recently:<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43200793">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43200793</a><p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43232410">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=43232410</a>

Looks like we are approaching the &quot;distributed&quot; phase of the distributed-centralized computing cycle :)<p>Not saying this is bad, but it&#x27;s just interesting to see after being in the industry for 8 years.

Isn’t the whole point of DuckDB is that it’s not distributed?

I&#x27;m not massively knowledgable about the ins and outs of DeepSeek, but I think I&#x27;m in the right place to ask. My understanding is DeepSeek:<p>- Created comparable LLM performance for a fraction of the cost of OpenAI using more off-the-shelf hardware.<p>- Seem to be open sourcing lots of distributed stuff.<p>My question is, are those two things related? Did distributed computing allow the AI model somehow? If so how? Or is it not that simple?

Does anyone have blogs with benchmarks to show the performance of running smallpond let alone 3fs + smallpond?<p>A lot of blogs praise these new systems, but don&#x27;t really provide any numbers :&#x2F;

spark is getting a bit long in the tooth.. interesting to see duckdb integrated with Ray for data-access partitioning across (currently) 3FS. probably a matter of time before they (or someone) supports S3. It should be noted that duckdb (standalone) actually does a pretty good job scanning s3 parquet on its own.