Moving a billion Postgres rows on a $100 budget

You can do something better for 0$, just install the TimescaleDB postgres extension and execute:<p>SELECT create_hypertable(&#x27;public.challenge_1br&#x27;, by_range(&#x27;time&#x27;));<p>Now, enjoy your better than Snowflake query performance performance at no extra cost.

I&#x27;m confused. Everything sounds very expensive to me.<p>The last table which compares it with the other vendors is surprising. Even Stich Data (cheapest) costs $1 to move 240K records: (1B &#x2F; 4,166.67 = 240K). Is this real?<p>So, their solution costs $1 to process 13.6M records. Sounds like this is not very share-worthy.<p>What I&#x27;m missing here?

&gt; Moving 1 billion rows is no easy task<p>This isn&#x27;t an accurate premise. Modern OLAP databases make dealing with billions to trillions of rows manageable, including on a single server. Exporting &quot;select * from table&quot; from an OLTP such as Postgres or MySQL into an OLAP is trivial and quite fast, and if 100M rows&#x2F;sec on commodity servers isn&#x27;t fast enough, there&#x27;s always performance tuning [1].<p>[1] <a href="https:&#x2F;&#x2F;altinity.com&#x2F;blog&#x2F;loading-100b-rows-in-minutes-in-altinity-cloud" rel="nofollow">https:&#x2F;&#x2F;altinity.com&#x2F;blog&#x2F;loading-100b-rows-in-minutes-in-al...</a>

How about 11b and a horrible python script over to parquet to your wee little NAS for your homelab?

I&#x27;ve already thought of the follow-up to this article, &#x27;Querying a billion rows on a $XX budget.&#x27; Let me give you my answer directly: switch from Snowflake to StarRocks. It&#x27;s an open-source project under the Linux Foundation, with speed that&#x27;s more than adequate, especially for queries involving multiple tables. If you&#x27;re interested, you might want to check it out, <a href="https:&#x2F;&#x2F;medium.com&#x2F;starrocks-engineering&#x2F;how-to-reduce-snowflake-costs-by-80-deb87aa69bc5" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;starrocks-engineering&#x2F;how-to-reduce-snowf...</a>.

If someone is price conscious, why move from postgres to snowflake?

Why not just set up a replica pgsql, then break the connection and upgrade it to primary?<p>It&#x27;s amusing how much effort is put into ETL these days. I remember when ETL departments were filled with the sludge of the programming world. It took ETL departments weeks to generate a CSV, and it would inevitably have massive numbers of errors because they didn&#x27;t actually follow the format that was specified on the form they forced everyone to use.

If someone asked me how much it would cost to move a billion Postgres rows, I would say &quot;probably under $100.&quot;<p>I just had to move 500 million &quot;rows&quot; into S3, and it came in at about $100. I would expect S3 to be more expensive.

(edit: mostly offtopic observation follows)<p>I only knew Snowflake the id selection algorithm, so was a bit confused, but googling &quot;snowflake db&quot; showed me this blurb and now I&#x27;m even more confused.<p>&gt; Snowflake enables organizations to learn, build, and connect with their data-driven peers. Collaborate, build data apps &amp; power diverse workloads in the ...

This was a disappointing article. It was expecting it to explore validating the integrity and consistency of the data, but that just seems to be handwaved away by a short section saying PeerDB handles it. This is especially disappointing since the article calls that out as one of the cruxes that makes this so difficult.

Does the elastic license fall under open source as defined by OSI?

&gt; Moving a Billion Postgres Rows on a $100 Budget<p>In a magical universe where your time is free.

Postgres seriously needs a columnstore backed table instead of just a rowstore.<p>MSSQL has this and it is magic. SingleStore has it, and it is wonderful.<p>I&#x27;m willing to give a bounty of $1000 to whoever adds that into main postgres tree.<p>Snowflake is great as a warehouse. it&#x27;s latency is shit when it comes to fast lookups and aggregates. If you can tolerate &gt;1s api calls, that is fine. It takes forever to insert a few rows in a large table.<p>If you want a proper live DB, snowflake is a rich man&#x27;s poor database.