TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Inserting 1.8M Rows/s from Pandas into QuestDB with Arrow, Rust and Cython

1 点作者 amunra__大约 2 年前

1 comment

amunra__大约 2 年前
Hi, I&#x27;m the original author of the QuestDB Python client library and benchmark.<p>It all started when we had one of our users needing to insert quite a bit of data into our database quickly from Pandas. They had a dataframe that took 25 minutes to serialize row-by-row iterating through the dataframe. The culprit was .iterrows(). Now it&#x27;s a handful of seconds.<p>This took a few iterations: At first I thought this could all be handled by Python buffer protocol, but that turned out to create a whole bunch of copies, so for a number of dtypes the code now uses Arrow when it&#x27;s zero-copy.<p>The main code is in Cython (and the fact that one can inspect the generated C is pretty neat) with supporting code in Rust. The main serialization logic is in Rust and it&#x27;s in a separate repo: <a href="https:&#x2F;&#x2F;github.com&#x2F;questdb&#x2F;c-questdb-client&#x2F;tree&#x2F;main&#x2F;questdb-rs">https:&#x2F;&#x2F;github.com&#x2F;questdb&#x2F;c-questdb-client&#x2F;tree&#x2F;main&#x2F;questd...</a>.