TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Ask HN: Good examples of fault-tolerant Erlang code?

147 点作者 roeles超过 1 年前
Hi HN! I&#x27;m trying to learn more about Erlang and how it achieves fault-tolerance. I am pretty up-to-date on the talks that Joe Armstrong gave, and I&#x27;ve read his thesis. This is already quite informative, but I wonder if there are any codebases out there which are good examples of fault-tolerance.<p>I&#x27;m particularly curious about when to split off a new process, and what &quot;if things fail, do something simpler&quot; means in practice.<p>Any suggestions?

12 条评论

toast0超过 1 年前
If you follow OTP design principles, you end up with a supervision tree, and a lot of code like...<p><pre><code> ok = do_something_that_might_fail() </code></pre> If it returns ok: great, it worked and you move on. If it doesn&#x27;t return ok, the process crashes, you get a crash report, and the supervisor restarts it, if that&#x27;s how the supervisor is configured. Presumably it starts properly and deals with future requests.<p>There&#x27;s two issues you might rapidly encounter.<p>1) if a supervised process restarts too many times in an interval, the supervisor will stop (and presumably restart), and that cascades up to potentially your node stopping. This is by design, and has good reasons, but might not be expected and might not be a good fit for larger nodes running many things.<p>2) if your process crashes, its message queue (mailbox) is discarded, and if you were sending to a process registered by name or process group (pg), the name is now unregistered. This means a service process crashing will discard several requests; the one in progress which is probably fine (it crashed after all), but also others that could have been serviced. In my experience, you end up wanting to catch errors in service processes, log them, and move on to the next request, so you don&#x27;t lose unrelated requests. Depending on your application, a restart might be better, or you might run each request in a fresh process for isolation... Lots of ways to manage this.
评论 #38804469 未加载
评论 #38805890 未加载
评论 #38805238 未加载
评论 #38802546 未加载
ramchip超过 1 年前
RabbitMQ is IMHO probably the best open source example tackling a large, complicated real world problem with graceful degradation (e.g. if a queue keeps crashing).<p>Elixir has a lot of smaller but very high quality libraries to learn from. You may be interested in how Ecto &amp; Postgrex manage DB connections, in particular how connection sockets are “borrowed” so data doesn’t get repeatedly messaged (read: copied) between processes. Bandit &#x2F; Thousand Island also make interesting decisions for process structure in HTTP1.1 vs HTTP2.<p>I think a common mistake is to create processes mimicking classic OOP structure, like an OrderProcessor, ShippingManager, etc. Processes in Erlang are a unit of fault tolerance, not code organization. This means more usually you’ll have one process per request, potentially calling code from many different modules; since requests are the things you want to fail separately from each other.<p>In RabbitMQ’s case for instance connections and queues are processes, but exchanges are not. It would feel natural to model the problem as three processes with messages going Connection -&gt; Exchange -&gt; Queue, but in reality an exchange is a set of routing rules that can be applied by a connection directly, which avoids a lot of complexity and overhead.<p>Last thing I’d note is supervision trees etc. are really about handling _unexpected_ errors (Joe uses the terms faults and errors with different meanings iirc). If you want a web request to be retried a few times with a delay, don’t use a supervisor for that, just loop with a sleep. Same for things like validating inputs from a form, usually you’d want to give the user a hint and not just crash.<p>Some other useful links:<p>- <a href="https:&#x2F;&#x2F;aosabook.org&#x2F;en&#x2F;v1&#x2F;riak.html" rel="nofollow">https:&#x2F;&#x2F;aosabook.org&#x2F;en&#x2F;v1&#x2F;riak.html</a> (bit old, but another large codebase)<p>- <a href="https:&#x2F;&#x2F;ferd.ca&#x2F;the-zen-of-erlang.html" rel="nofollow">https:&#x2F;&#x2F;ferd.ca&#x2F;the-zen-of-erlang.html</a><p>- <a href="https:&#x2F;&#x2F;www.theerlangelist.com&#x2F;article&#x2F;spawn_or_not" rel="nofollow">https:&#x2F;&#x2F;www.theerlangelist.com&#x2F;article&#x2F;spawn_or_not</a>
octacat超过 1 年前
The simple answer: supervision trees. And fault-tolerance usually means that the failing process would be restarted. It would not handle stuff like netsplits or node going down though.<p>Check code of cowboy, ejabberd, MongooseIM, RabbitMQ for examples. There are many factors on decision when to make a new process. Data locality, the pattern of interaction with other processes, performance considerations. Good idea is to have one process per TCP connection, but not one process per each routed message. And be careful with blocking gen_server calls - these could block or fail.
评论 #38804134 未加载
评论 #38806874 未加载
评论 #38802552 未加载
al2o3cr超过 1 年前
Step zero is definitely the OTP Design Principles doc (part of the OTP distribution):<p><a href="https:&#x2F;&#x2F;www.erlang.org&#x2F;doc&#x2F;design_principles&#x2F;users_guide" rel="nofollow">https:&#x2F;&#x2F;www.erlang.org&#x2F;doc&#x2F;design_principles&#x2F;users_guide</a><p>There are some good texts that have more examples:<p>Erlang &amp; OTP in Action - <a href="https:&#x2F;&#x2F;www.manning.com&#x2F;books&#x2F;erlang-and-otp-in-action" rel="nofollow">https:&#x2F;&#x2F;www.manning.com&#x2F;books&#x2F;erlang-and-otp-in-action</a><p>Designing for Scalability with Erlang&#x2F;OTP - <a href="https:&#x2F;&#x2F;www.oreilly.com&#x2F;library&#x2F;view&#x2F;designing-for-scalability&#x2F;9781449361556&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.oreilly.com&#x2F;library&#x2F;view&#x2F;designing-for-scalabili...</a><p>One big example of distributed Erlang is Riak:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;basho&#x2F;riak">https:&#x2F;&#x2F;github.com&#x2F;basho&#x2F;riak</a>
评论 #38804310 未加载
评论 #38794265 未加载
评论 #38807770 未加载
评论 #38802564 未加载
chadd超过 1 年前
I have written a lot of Erlang code over the years, including an Erlang Redis clone which had some interesting performance characteristics[1] ... though not recently, I went too far down the engineering management track at Snap and elsewhere... but I worked closely with Fernando &quot;El Brujo&quot;[2] when he was CTO of my consultancy. If you want to see beautiful, canonical Erlang code, he&#x27;s still slinging it out. Dig through his repos on Github, or better yet, ask him to provide his suggestions.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;cbd&#x2F;edis">https:&#x2F;&#x2F;github.com&#x2F;cbd&#x2F;edis</a> [2] <a href="https:&#x2F;&#x2F;github.com&#x2F;elbrujohalcon">https:&#x2F;&#x2F;github.com&#x2F;elbrujohalcon</a>
thibaut_barrere超过 1 年前
If you were to consolidate all the info (including links published by Joe) into an informative blog post, it could become the “2024 reference bookmark” for a lot of people.<p>I have thought of writing this! It would be quite useful to a lot of people.
评论 #38813958 未加载
ihuk超过 1 年前
You don&#x27;t achieve fault tolerance solely by using Erlang. Erlang does not inherently &#x27;achieve fault tolerance.&#x27; Instead, you make your system fault-tolerant through deliberate engineering. While Erlang provides tools and design guidelines, the responsibility for achieving fault tolerance ultimately lies with you. Source: I implemented and operated a large Erlang system for approximately 3 years.
评论 #38804103 未加载
评论 #38804113 未加载
评论 #38808488 未加载
asa400超过 1 年前
&gt; I&#x27;m particularly curious about when to split off a new process, and what &quot;if things fail, do something simpler&quot; means in practice.<p>Processes are failure and concurrency barriers.<p>Failure: one process crashing does not crash another process, unless you explicitly want it to (e.g., via Erlang&#x27;s `link` functionality). So, if you have multiple operations that must not interfere with each other in the case of one of them misbehaving (e.g., your application makes multiple HTTP requests in parallel), you want them in separate processes.<p>Concurrency: processes are independently and preemptively scheduled by the VM. If you have multiple operations that are not necessarily sequentially ordered, and you want to run them at the same time, you put each of them in a process. One example problem where this applies would be the handling of incoming TCP messages, where each message is not related to the previous or subsequent messages, and you want to be able to process multiple messages at the same time.<p>If you handle each new message in its own process, the VM will schedule the processing of those messages such that the processing of one message will not interfere with the processing of another. It accomplishes this by tracking a rough proxy of CPU time each process uses (called &quot;reductions&quot; in Erlang) and descheduling processes that consume too many resources and giving other processes a chance to run for a bit. (Note that this is just one example and ignores any performance considerations. There are other approaches but I am omitting them for simplicity&#x27;s sake)<p>There are a number of good libraries to look at for these in practice. I&#x27;d personally go look at Cowboy and&#x2F;or Ranch as they deal with lots of IO. Oban is an Elixir job queue library that is fantastic and has very high code quality. Another good one would be Poolboy, which is a worker pool library.
brudgers超过 1 年前
<i>when to split off a new process</i><p>Always?<p>Times some factor so you have several instances of the same thing in case one fails.<p>Good luck.
jmnicolas超过 1 年前
I don&#x27;t know Erlang so take it with a grain of salt, but maybe take a look at the CouchDB code base?<p>It&#x27;s a NoSQL DB written in Erlang. I looked at it a few years ago, its master to master replication seemed cool.
rramadass超过 1 年前
Not code but an excellent presentation of design&#x2F;architecture of a Real-World fault-tolerant and distributed System in Erlang&#x2F;Elixir - <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=pQ0CvjAJXz4" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=pQ0CvjAJXz4</a>
vladimirralev超过 1 年前
My advise is don&#x27;t take Erlang&#x27;s fault-tolerance promises too seriously. It&#x27;s just a little framework that helps in some cases and gets in the way in other cases.<p>I&#x27;ve seen many Erlang systems fail in funny ways, including some of the big examples given here. Supervision trees are cool but it&#x27;s clearly nonsense to hardcode restart strategy and timing numbers for workers as if all failure modes are the same and deployed in the same network&#x2F;capacity&#x2F;resource&#x2F;conditions with any number of workers. The strategy and schedule for recovering 10 crashed resource workers will clearly be different when you have 1M workers. The strategy will be different if you are timing out on network or if you are getting a resource error and have better things to do than restarting workers.<p>Focus on fault-tolerance outside erlang - have standby capacity in isolation and load-balance properly, shard the system in isolated pieces as much as you can.