TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Zero downtime Postgres upgrades

389 点作者 brentjanderson超过 1 年前

19 条评论

aeyes超过 1 年前
There is a better way than fully copying table content one by one which is very I&#x2F;O heavy and will not work if you have very large tables.<p>You can create a replication slot, take a snapshot, restore the snapshot to a new instance, advance the LSN and replicate from there - boom, you have a logical replica with all the data. Then you upgrade your logical replica.<p>This article from Instacart shows how to do it: <a href="https:&#x2F;&#x2F;archive.ph&#x2F;K5ZuJ" rel="nofollow noreferrer">https:&#x2F;&#x2F;archive.ph&#x2F;K5ZuJ</a><p>If I remember correctly the article has some small errors but I haven&#x27;t done this in a while and I don&#x27;t exactly remember what was wrong. But in general the process works, I have done it like this several times upgrading TB-sized instances.
评论 #38623425 未加载
评论 #38621730 未加载
评论 #38634575 未加载
评论 #38635003 未加载
评论 #38623652 未加载
natbennett超过 1 年前
The approach here is interesting and well-documented! However, this line gives me pause—<p>&gt; Modern customers expect 100% availability.<p>This is not my preference as a customer, nor has it been my experience as a vendor. For many workloads consistency is <i>much</i> more important than availability. I’m often relieved when I see a vendor announce a downtime window because it suggests they’re being sensible with my data.
评论 #38621118 未加载
评论 #38622966 未加载
CubsFan1060超过 1 年前
AWS supports blue green deployments now. <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;about-aws&#x2F;whats-new&#x2F;2023&#x2F;10&#x2F;amazon-rds-blue-green-deployments-aurora-rds-postgresql&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;aws.amazon.com&#x2F;about-aws&#x2F;whats-new&#x2F;2023&#x2F;10&#x2F;amazon-rd...</a>
评论 #38621672 未加载
评论 #38620458 未加载
shayonj超过 1 年前
This is great! I wrote a tool that automates most of the things you came across. If you find it useful or would like to extend it with your feedback&#x2F;ideas, I&#x27;d love to have them! Thanks for sharing<p><a href="https:&#x2F;&#x2F;github.com&#x2F;shayonj&#x2F;pg_easy_replicate">https:&#x2F;&#x2F;github.com&#x2F;shayonj&#x2F;pg_easy_replicate</a>
评论 #38621463 未加载
vasco超过 1 年前
&gt; No amount of downtime - scheduled or otherwise - is acceptable for a service like Knock<p>doubt.jpeg<p>If you have a complex system, you have incidents, you have downtime. A 15min downtime window announced in advance is fine for approximately 100% of SaaS businesses. You&#x27;re not a hospital and you&#x27;re not the power station. So much fake work gets done because people think their services are more important than they are. The engineering time you invested into this, invested into the product, or in making the rest of your dev team faster, would&#x27;ve likely made your users much happier. Specially if you can queue your notifications up and catch up after the downtime window.<p>If you have enterprise contracts with SLAs defining paybacks for 15min downtime windows, then I guess you could justify it, but most people don&#x27;t. And like I mentioned, you likely already have a handful of incidents of the same or higher duration in practice anyway.<p>This is specially relevant with database migrations where the difference in work to create a migration of &quot;little downtime&quot; to &quot;zero downtime&quot; is usually significant. In this case though, seeing as this was a one time thing (newer versions of PostgreSQL on RDS allow it out of the box) it is specially hard to justify in my opinion, as opposed to if this was going to be reused across many versions or many databases powering the service.
评论 #38620797 未加载
评论 #38621651 未加载
评论 #38621665 未加载
评论 #38622335 未加载
评论 #38621244 未加载
评论 #38624136 未加载
评论 #38625101 未加载
评论 #38621678 未加载
评论 #38621318 未加载
评论 #38622012 未加载
T-Winsnes超过 1 年前
We&#x27;re going through this right now with hava.io<p>AWS RDS postgress 11.13 &gt; 15.5<p>We ended up going with a relatively straight forward approach of unidirectional replication using pglogical. We have some experience doing the same migration from Google Cloud sql to AWS rds with zero downtime as well, which made us pretty confident that this will work and not impact customers in any visible way.<p>pglogical makes it relatively straight forwards to do this kind of migration. It&#x27;s not always fast, but if you&#x27;re happy with waiting for a few days while it gradually replicates the full database across to the new instances.<p>For us it gave us a bit more freedom in changing the storage type and size which was more difficult to do with some of the alternative approaches. We had oversized our storage to get more iops, so we wanted to change storage type as well as reducing the size of the storage, which meant we couldn&#x27;t do the simple restore from a snapshot.
AtlasBarfed超过 1 年前
Oh you mean like aws promised us during the &quot;sales engineering&quot; but failed to deliver when a major version upgrade was forced upon us?
dboreham超过 1 年前
Surprised you can&#x27;t initialize a replica from a backup. That would have saved all the farting around streaming the old stable DB content to the new server.<p>Also, this isn&#x27;t &quot;zero downtime&quot; -- there&#x27;s a few seconds down time while service cuts over to the new server.<p>The article omits details on how consistency was preserved -- you can&#x27;t just point your application at both servers for some period of time, for example. Possibly you can serve reads from both (but not really), but writes absolutely have to be directed to only one server. Article doesn&#x27;t mention this.<p>Lastly, there was no mention of a rollback option -- in my experience performing this kind of one-off fork lift on a large amount of data, things sometimes go off the rails late at night. Therefore you always need a plan for how you can revert to the previous step, go to bed with the assurance that service will still be up in the morning. Specifically that is hard if you&#x27;ve already sent write transactions to the new server but for some reason need to cut back to the old one. Data is now inconsistent.
评论 #38628874 未加载
tehlike超过 1 年前
The sequence thing is definitely interesting, I stopped using them a while ago, using mostly sequential uuid (or uuid v7), or use something like HiLo <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Hi&#x2F;Lo_algorithm" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Hi&#x2F;Lo_algorithm</a>
评论 #38622555 未加载
评论 #38622411 未加载
ohduran超过 1 年前
Not to downplay the absolute behemoth of a task they manage to pull out successfully...but why not upgrading as new versions came along, with less fanfare?<p>It is a great read, but I can&#x27;t shake the feeling that it&#x27;s about a bunch of sailors that, instead of going around a huge storm, decided to go through it knowing fully well that it could end in tragedy.<p>Is the small upgrades out of the question in this case? As in &quot;each small one costs us as much downtime as a big one, so we put it off for as long as we could&quot; (they hint at that in the intro, but I might be reading too much into it).
评论 #38620483 未加载
评论 #38621434 未加载
评论 #38620375 未加载
oopsthrowpass超过 1 年前
There is no problem in distributed systems that can&#x27;t be solved with a well placed sleep(1000) :D<p>But anyway, good job, Postgres is quite a DBA unfriendly system (although better than it used to be still pretty bad)
TechIsCool超过 1 年前
With the mention of AWS RDS and Aurora, I am curious if you had thought about creating a replication slot, adding a replica to the cluster and then promoting the replica to its own cluster. Then connecting the new cluster to the original with the replication slot based on the position of the snapshot. This would save the large original replication time and also keep the sequences consistent without manual intervention.
评论 #38621004 未加载
fosterfriends超过 1 年前
Great write up y’all! Writing this detailed of a post isn’t easy, and it works to build confidence in your technical prowess. Keep up the great work :)
wihoho123超过 1 年前
The title is very misleading given it still has downtime for at least several seconds.<p>Also, it&#x27;s super tedious work, and mistakes could happen during any step. Lastly, this updage is deeply coupled with application logic. Already feel a pain.<p>Why don&#x27;t you just use aurora, then it&#x27;s 0-downtime going forward?
评论 #38643156 未加载
whalesalad超过 1 年前
Another epic win for the BEAM!
评论 #38621450 未加载
andrewmcwatters超过 1 年前
Uh... How big was their database? Did I miss it? I don&#x27;t think they said.
评论 #38621706 未加载
hurtbird超过 1 年前
Anyone interested in having an all-in-one tool that does zero-downtime Postgres updates where the only input is database credentials? Nothing to setup manually.
october8140超过 1 年前
Heroku just does this. At my old job we would scale the database using replication multiple times a week depending on expected traffic.<p><a href="https:&#x2F;&#x2F;devcenter.heroku.com&#x2F;articles&#x2F;heroku-postgres-follower-databases" rel="nofollow noreferrer">https:&#x2F;&#x2F;devcenter.heroku.com&#x2F;articles&#x2F;heroku-postgres-follow...</a>
评论 #38620066 未加载
评论 #38620212 未加载
throwawaaarrgh超过 1 年前
&gt; Postgres sits at the heart of everything our systems do.<p>Did the people making these decisions never take Computer Science classes? Even a student taking a data structures module would realize this is a bad idea. There&#x27;s actually more like two dozen different reasons it&#x27;s a bad idea.
评论 #38620365 未加载
评论 #38620566 未加载
评论 #38620356 未加载