TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

How Fast Can We Write a 10M Row CSV to a Postgres Database with Pandas?

3 pointsby otacustabout 3 years ago

1 comment

data_dan_about 3 years ago
I wrote this article after a colleague pointed out that the Pandas DataFrame.to_sql() method uses row-by-row INSERTs. There are plenty of good reasons for this, and the to_sql method works great with many different SQL database flavors, but it&#x27;s not fast.<p>This article compares the performance of different methods for writing a Pandas DataFrame to a PostgreSQL database using the to_sql method on DataFrames ranging from 100 rows to 10,000,000 rows.