Would be nice to know why RabbitMQ/ZeroMQ didn't fit the bill [scale/speed?] -- I can somewhat understand the not relying on SQS but then there is anyways a reliance on EC2 right ?
I would be interested if anybody could do a comparison between that PinLater and Celery. Feels like there is a bit of NIH involved, unless it's the 100K/sec rate is not something that celery could realize and PinLater can.
Thanks all for the comments about the PinLater post - great to see this discussion! Wanted to clarify a few things:<p>* Yes, we did evaluate a few open source solutions before deciding to build in-house. We couldn't at the time find any solution that met all the requirements that we had: scale, scheduling jobs ahead of time, transactional ACKs, priority queues, configurable retry policies, great manageability (e.g. ability to rate limit a queue online, inspect running jobs, review failed job stack traces). [update: also support for clients/jobs in multiple languages: Python, Java, Go, C++]<p>* Note that we were building a v2 system. Not improving substantially on the existing system on multiple dimensions would not quite make it worth the cost of doing large scale migration of tasks. We <i>had</i> to meet all the requirements, could not compromise on missing a few.<p>* Sorry I didn't mention explicitly in the post, but we <i>do</i> plan to open source PinLater, hopefully later this year.<p>Happy to answer any other questions or specifics!
Gearman is a very simple queue that is even easier than RabbitMQ (in my opinion). You create worker threads, and submit jobs to the server. The server sends the job to a worker thread, handles re-queuing it if it fails, etc..<p>Basically these guys 'invented' the queue.<p>Also, the problem of acknowledging success without blocking can be solved without queues - by programming in async environments like NodeJS
Sounds like a perfect fit for beanstalkd[1]. It has scheduled jobs, explicit ACKs, priority and some other nice features (time-to-run, write ahead logging, job burying/kicking, queue pausing).<p>We have been using for years to process 1+ million/jobs per day without a single issue.<p>[1] <a href="http://kr.github.io/beanstalkd/" rel="nofollow">http://kr.github.io/beanstalkd/</a>
The ZeroMQ position on why brokers like this may not be a good idea: <a href="http://zeromq.org/whitepapers:brokerless" rel="nofollow">http://zeromq.org/whitepapers:brokerless</a>