I've built quite a few apps that use queueing to do the heavy processing behind the scenes, and I've always used a polling system to check progress. But it always has felt like a half-baked solution.<p>Are there any patterns or documentation projects or books or tutorials or anything else out there that cover how to have users submit something into a queue system, provide immediate feedback, then provide accurate progress updating as the job is processed, then show the results, all without having the user refresh their process or navigate away and back again?<p>Let's say for instance, I built a service that lets a user upload 20 images at a time to be cropped and resized. Is there a commonly accepted pattern for alerting them of the resizing job progress as it is happening?
The implementation/operational complexity of a queue (assuming you don't just hammer it into your existing database, which can sometimes be a bad idea) many times outweighs the user having to wait an extra 200ms.<p>It's nice when you <i>have</i> to have a queue for image/audio/video processing, page scraping, etc because then you can use the queue for other stuff, but for the most part, excluding large jobs, you can get away without having a queue for quite a while.<p>So if you have a queuing system, great, use it. If you don't, try to avoid building it until you really need it.
I'm currently working with ActiveMQ. I try to think of it as multiple IRC networks. Channels as topics for users by users, servers as gateways to isolated materials/users, server global messages as topics for users by servers, server local messages as queues for users by servers and private messages as queues for users by users.<p>It is absolutely fascinating offloading a message to the queue and letting the back-end handle workflow and shuffling it to another queue for micro-tasking (such as in the article). In fact, the topic/queue paradigm has really shifted my whole view on services.<p>Forgot to mention the queue added a layer of security. Our front-end is now the same as spawning a shell in the server (as far as data access). If you can spawn a shell on the webserver, the server is isolated from everything except the MQ ip/port combo. This means that what was possible on the front end is the only thing possible on the back end (as far as penetrating data). I'm sure a clever person could get figure out a way, but for your average person, that will be a show stopper.
I understand the purpose of doing this, but in most cases I don't see any extremely simple way to implement it. Lots of decisions tend to revolve around whether the development/maintenance time spent doing something justifies the efficiency provided by it.<p>Especially if you're building a minimum viable product, if something on a server responds 10ms later than a queue implementation, but takes 1 hour to implement instead of 5, I am inclined to go with the 10ms delay.
Something at work we use that's great for this is ResqueDelayable. Typically I see Resque tutorials extract bits of logic out into worker classes, but this adds refactoring/organization overhead. ResqueDelayable allows this non-queued work:<p><pre><code> def post_comment
add_comment
self.notify_followers # Could have lots of followers
Analytics.record_new_comment # Unnecessary for api response
end
</code></pre>
to become queued work, with no real refactoring:<p><pre><code> def post_comment
add_comment
self.rdelay.notify_followers
Analytics.rdelay.record_new_comment
end</code></pre>
Makes me think of Meteor where changes to a document are reflected in the client immediately and then possibly rolled back if the the server-side equivalent change fails which typically would be very rare.