TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Easy cluster parallelization with ZeroMQ

64 点作者 mdup将近 10 年前

5 条评论

gh02t将近 10 年前
Worth mentioning that the cluster tools in IPython implement basically the same system as he ends up with, plus lots of other functionality and it is using ZMQ as the backend as well. If you&#x27;ve got a small to medium sized task (I&#x27;d say less than 500 nodes is a good number) to run on your cluster ipcluster is pretty great. Even when I have a proper queuing system like SLURM or PBS in place, I still often use ipcluster on top.<p><a href="http:&#x2F;&#x2F;ipython.org&#x2F;ipython-doc&#x2F;dev&#x2F;parallel&#x2F;" rel="nofollow">http:&#x2F;&#x2F;ipython.org&#x2F;ipython-doc&#x2F;dev&#x2F;parallel&#x2F;</a>
评论 #9614346 未加载
评论 #9616013 未加载
avargas将近 10 年前
ZeroMQ is amazing. A few years ago I built a prototype project for a client that basically was fail2ban but scalable. It monitored nginx logs and broadcasted some information, and workers did the rest. Most of the data was in memory and passed around, and communication was done via ZeroMQ. It was done this way so that we could split the heavy-load components off the server and into workers, and allow the server to simply do it&#x27;s job: tail nginx logs and act upon ban requests from workers. It was amazing, sadly I never completed it and deployed it on a production environment but from initial tests, it outperformed fail2ban by a lot.
Merkur将近 10 年前
I had fun reading - thx. I found the solution of &quot;requesting workers&quot; interesting. My first reflex to the round robin argument was - a better load balancer but the requesting worker made that totally not necessary! I wounder what negative sites sutch architectures may have?
nickpsecurity将近 10 年前
Nice article. I hope to see a benchmark of it on a less parallel problem with low-latency network hardware. Maybe a comparison of that against a vanilla MPI or parallel framework. We&#x27;ll get to see what its real overhead is.
评论 #9614571 未加载
FullyFunctional将近 10 年前
Great writeup. Perhaps slightly less clean, you could cut the messages in half by having the available report the result as well (empty in the initial case).<p>Is the worker idling between replying and receiving new work? In other words, does ZMQ enable overlapping, say always having two outstanding requests per worker?