TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Dev / Stage / Prod is the wrong pattern for data pipelines

39 pointsby merinidover 1 year ago

4 comments

SOLAR_FIELDSover 1 year ago
Yet another discovery that ephemeral environments are the more robust way to design things. Yes they are harder to setup and require some proper thought and engineering but they end up paying themselves off down the line many times over.
评论 #37267386 未加载
评论 #37244417 未加载
NBJackover 1 year ago
This doesn&#x27;t scale well. It is a perfectly fine approach for smaller systems with a few dependencies, but you are going to have serious headaches whenever you (1) start to see more complex internal system dependencies, and&#x2F;or (2) start taking on deeper integration with external systems like cloud infrastructure, other services, etc. Once you hit this inflection point, you either start getting very robust with your integration boundaries (and likely developing more complex &#x27;stubs&#x27;), or go the dev&#x2F;stg&#x2F;prod route.<p>You start with a database? Great. But wait, you need bulk storage now, so you start sticking it in a cloud bucket (and ensure you use a separate namespace for it). But then Team 2 introduced a new service you now need to spin up in a separate container, so you pull their repo. Then there&#x27;s a production issue that could have been solved by proper AB testing, so you decide to go with a third party solution that offers that. The party continues, and soon your simple one-click setup ends up so complicated you end up with a full time person just keeping it alive. Whoops! Someone got the cloud namespace wrong on their desktop instance, and production data got hosed. Etc.
评论 #37253933 未加载
评论 #37267490 未加载
erhaetherthover 1 year ago
Sounds like a complicated way of saying every developer should have their own DB instead of a shared dev instance.
评论 #37267585 未加载
Raminj95over 1 year ago
Is there some more examples or blog posts that talk about this? I find the idea interesting and possibly very applicable in my work but just from this post alone I don’t feel like I have grasped it well enough to implement this.
评论 #37267692 未加载