Asynchronous Iteration Patterns (in Node.js)

Yes, this is necessary. Now, build up a few more of these patterns, start watching them intertwine with each other in ways that require you to manually interleave them, come to the realization that building your glorious asynchronous stack on top of a fundamentally synchronous language was a bad idea.<p>In languages that are not fundamentally synchronous, such as Erlang, you do not leap through hoops to manage this. You simply write a function that performs your insertions in the straightforward and obvious way, and the runtime manages it with no blocking at any point.<p>I actually don't like the frequently-made assertion that "a pattern is automatically a weakness in the language", but it does apply here. As the Node.js community laboriously builds up the patterns necessary to work in this paradigm, recapitulating the work done in numerous other async-on-top-of-sync-language libraries, it probably is worth keeping the assertion in mind. You shouldn't even have to <i>think</i> about this, let alone argue which way is the best way to do it.<p>(Somebody modded this down. Maybe I should make it clear that I'm actually speaking from experience on another asynchronous project written in Perl on top of glib's asynchronousness, which isn't fundamentally different from Node.js'. I'm not speculating, I've been on the receiving end of this complexity explosion. It <i>will</i> happen.)

This is a great demonstration of the fundamental issue with node. His code goes from:<p><pre><code> function insertCollection(collection) { for(var i = 0; i &#60; collection.length; i++) { db.insert(collection[i]); } } </code></pre> To this:<p><pre><code> function insertCollection(collection, callback) { var coll = collection.slice(0); // clone collection (function insertOne() { var record = coll.splice(0, 1)[0]; try { db.insert(record, function(err) { if (err) { callback(err); return } if (coll.length == 0) { callback(); } else { insertOne(); } } } catch (exception) { callback(exception); } })(); } </code></pre> Hopefully this article will work as an eye-opener for anyone who was not yet convinced that node badly needs a concurrency abstraction.

You'd overflow your stack after inserting 600-700 records:<p><a href="https://gist.github.com/845874" rel="nofollow">https://gist.github.com/845874</a>

I posted this as a comment on the blog post, though I would share it here.<p>This is awesome that you wrote a post on this, it is an important building block to proper node.js coding. I know I had trouble with it.<p>There is another way to handle asynchronous iteration, while similar to what you have done, gives a bit more flexibility and is abstracted out so you can use it anywhere.<p><a href="https://gist.github.com/b5af7369ec9939ab7d94" rel="nofollow">https://gist.github.com/b5af7369ec9939ab7d94</a><p>This is using code from a script of mine as an example, but it should be pretty easy to see the pattern.<p>Here is your above serialized example with the above function I just showed you:<p><a href="https://gist.github.com/26bda51358610667f9f3" rel="nofollow">https://gist.github.com/26bda51358610667f9f3</a><p>---<p>On a side note, I wanted to say that people hell bent on turning node.js into another language should just use that language. There is nothing wrong with the way node.js handles asynchronous code, and really keeping everything asynchronous instead of building in stuff to fake synchronous code is bad for node.js and the whole ecosystem. It promotes bad habits and bad code. If you can't handle callbacks or events, then maybe you should go back to using whatever you are more comfortable with, or write your own abstractions.

Here's how this looks in twisted (handling exceptions, capturing (but ignoring) outputs, etc...):<p><pre><code> @defer.inlineCallbacks def insertCollection(collection): for ob in collection: yield db.insert(ob) </code></pre> ...as much as people talk about the complexity of twisted with respect to async programming, they've really figured it out over the many years of the life of the project.<p>I agree with substack that you can do this without changing the language, but you do have to get something usable to people at some point.<p>I think most people who write any code in node.js run into this issue. I am a casual user, but ran into it a couple of days ago and did some research to find that there are many third-party modules I could bring in to do a for loop, but nothing built-in. The old way of doing this in twisted (manually managing deferreds) really turned a lot of people off. By the time twisted got inlineDeferreds and inlineCallbacks, people already had their impressions of how difficult things are.<p>I still think node.js is fun and viable, but it does need a for loop.

Interesting article - I really liked the updated example at the very end, demonstrating a new pattern (for me) for handling exceptions that get thrown by wrapping the DB inserts in their own function.

"Here we are using tail recursion to keep inserting the records."<p>Correct me if I'm wrong, but I don't think he means tail recursion. It's recursion, yes. But does tail recursion even make sense in an asynchronous setting?

Abstractions are absolutely necessary imo. Code like that is too long to be easily maintainable.<p><a href="http://groups.google.com/group/nodejs/browse_thread/thread/c334947643c80968/8d9ab9481199c5d8?show_docid=8d9ab9481199c5d8" rel="nofollow">http://groups.google.com/group/nodejs/browse_thread/thread/c...</a><p>Also, that guy is responsible for nodetuts, which I'm very thankful for!<p><a href="http://nodetuts.com/index.html" rel="nofollow">http://nodetuts.com/index.html</a>