A Fast, Simple, Queue Built on MongoDB

Developer aversion to learning new tools is astounding sometimes. Background workers and message queues have to be the most reinvented wheel of all time. 99% of the time, something free and great will already do what you want, and they've thought of things you haven't. Why not redirect some of this energy into using a tool like Celery[1]? Surely it's simpler to learn a tool than it is to build a new one from scratch?<p>* What happens when job's crash, fail, or get stuck?<p>* What about when the machines running the workers die or are isolated from the network?<p>* What happens if the backing store dies?<p>* How do you start, stop, restart, grow, and shrink your worker pool?<p>[1] <a href="http://ask.github.com/celery/getting-started/introduction.html" rel="nofollow">http://ask.github.com/celery/getting-started/introduction.ht...</a>

Perhaps I'm assuming too much, but it looks like you might be confusing <i>notification</i> with <i>queueing</i>. As you found with SQS, queueing systems can make for poor notification mechanisms. Did you try using AWS SNS (<a href="http://aws.amazon.com/sns/" rel="nofollow">http://aws.amazon.com/sns/</a>) for your first two use cases? We have similar notification use-cases internally, and SNS has worked great.

Doing a queue system where the workers wait in a busy-loop is pretty insane.<p>I'm saying that as the author of Delayed::Job so i'm pretty much responsible for those shenanigans.