It's a bit of a wobbly tower of reasoning to causally go from "you used node.js" to "you overwhelmed your cloud providers DNS service and not even dnsmasq can dig you out".<p>There are a ton of interventions you could do on that journey and I don't think it makes a convincing argument against picking X in the first place.<p>I spose it is a good example of unforeseen resource constraints that big successful systems encounter. In my limited experience almost all commonly used software breaks under massive load and you have to radically modify or write your own at the largest scales. But by then you are rich so it doesn't matter
Don't you solve at least half of this by putting a load balancer in front the service being hit (which is internal, so we control it), then the clients don't need to know which machines are up and working, you don't overload DNS packets with a bunch of endpoints, and the load balancer takes care of keeping track of which machines are available, and the clients don't need to share any state about the status of the individual remote machines.<p>I'm not sure what this piece is getting at ("don't do this thing that no one would do anyway"), other than ignoring 20+ years of well-understood scaling methodology, and listing a bunch of unrelated gotchas (at high enough traffic rates, you have to deal with running out of ephemeral ports even without the design outlined). This isn't a great modern example of the compounding effects of bad design.
I guess the obvious answer to this is: congratulations! You must have a successful product, or you wouldn’t be seeing so many requests. Now that you have a successful product, you have the budget to generate a solution.<p>Personally, in this situation, I would find some multithreaded RPC proxy thingy where a single instance would sit between the worker processes and the rest of the services.
The multi threaded one process app might work well - until you need to scale across many machines, then the multi-process architecture might be easier to scale. But you probably end up rewriting the whole architecture anyway.
This reads like a Hollywood disaster movie. Could it happen, okay? How likely is it given that enough go down the path of X and become large enough to be hitting these limits? I want to read <i>those</i> stories not this.