Introducing a new data structure, streams, in Javascript

As someone who conjured this exact thing up in another language after reading SICP, I have to suggest that the first two commenters here missed the point: this is a very basic but powerful structure that implements a lazy stream and is nearly a monad (many standard options are written for you but could be generalized).<p>Want it to apply to data I/O? Stick your reader into the HEAD function, store the pointer to the next byte in TAIL, and implement functions for reading X number of bytes and converting them into their new types.<p>Yes, you can do math with it. You can also implement process/procedure into it, if you consider yourself popping lambdas off the stream, and which lambda is Next can be guided by choices in the current lambda.<p>Mostly, you can use this device to hold any collection of things and operate upon them in a streamy-syntax kind of way. For those who are familiar with jQuery, it is a bit of a reimplementation of a lot of jQuery's basic array-data handling with extra helpers like "range" tossed in and a stronger focus on the laziness. Where jQuery's prime target is work with the DOM, this stream package's target is the more theoretical version of some of jQuery's features, specifically those that iterate over the contained arrays. You could wrap jQuery in this and get an amazing hybrid.<p>Don't sell it short; it really is another useful and powerful way to approach data management. Check out how you construct your own stream with this package, and think beyond the basic numeric approaches.

This comment thread reminds me a bit of "The Emperor's new clothes". I think the code for sieve (even though I understood what it did) is way too unnecessarily confusing and I would not want to see it in a project I'm on.<p>However, I do appreciate the usefulness of lazy evaluation. Still, I think this documentation could benefit by not making outlandish claims ("it will change the way you think") and just saying the author has implemented lazy evaluation in javascript (and maybe link to the wikipedia article on lazy evaluation and this stack overflow to explain why it's useful: <a href="http://stackoverflow.com/questions/2151226/what-are-the-advantages-of-lazy-evaluation" rel="nofollow">http://stackoverflow.com/questions/2151226/what-are-the-adva...</a>)

I was first introduced to the concept of 'streams' akin to this in SuperCollider, which uses them as building blocks for musical patterns.<p><a href="http://danielnouri.org/docs/SuperColliderHelp/Streams-Patterns-Events/Pbind.html" rel="nofollow">http://danielnouri.org/docs/SuperColliderHelp/Streams-Patter...</a><p><a href="http://danielnouri.org/docs/SuperColliderHelp/Streams-Patterns-Events/Pattern.html" rel="nofollow">http://danielnouri.org/docs/SuperColliderHelp/Streams-Patter...</a><p>SuperCollider is a fun language to study. You end up with compositional structures like the following (notice 'inf', which indicates an infinite loop)<p><pre><code> Pbind( \octave, 4, \degree, PstepNadd( Pseq([1, 2, 3]), Pseq([0, -2, [1, 3], -5]), Pshuf([1, 0, 3, 0], 2) ), \dur, PstepNadd( Pseq([1, 0, 0, 1], 2), Pshuf([1, 1, 2, 1], 2) ).loop * (1/8), \legato, Pn(Pshuf([0.2, 0.2, 0.2, 0.5, 0.5, 1.6, 1.4], 4), inf), \scale, #[0, 1, 3, 4, 5, 7, 8] ).play;</code></pre>

Um, 'new data structure'? You have to be kidding. This is implemented in practically every single functional programming language. Also, this was already done in JavaScript: <a href="https://github.com/Gozala/streamer" rel="nofollow">https://github.com/Gozala/streamer</a> and looks surprisingly similar to your library. As you can see, streams in JavaScript are nothing new.

First of all this is very nice. (I'm going to get a little negative shortly; I want to be clear that I'm not putting down the writer's efforts at all.)<p>Several commenters have compared these streams to Python generators. Yes, the two are similar. But there are are a couple of important things Python generators have, that these don't.<p>(1) Python generators, along with lists, tuples, files, net connections, etc., all support the standard iterator protocol. This means that, for many purposes, you don't need to know or care which kind of data source you are reading from. The itertools functions can be applied to all of them, etc. In contrast, these JS streams are dealt with via specialized member functions. You can't just take array-based code, pop in a stream, and expect it to work.<p>(2) The iterator returned by a Python generator is a wrapper around a single running computation, which can spit out multiple values. Retrieving a new value is not, at heart, a function call, but rather the continued execution of a coroutine that has already started. This has many advantages; for example, we can write a Python generator that produces an infinite dataset, using a simple loop. I don't think that works here.<p>Now, as for (1) above, I think this is something we could do in JS. It would just be a matter of everyone agreeing to support a particular interface.<p>OTOH, it seems to me that JS does not have the functionality to support (2). Python does. So do Ruby, Haskell, and Go -- each in its own way. But JS does not, AFAIK. Am I wrong? And should we be building support for (2) into JS and other languages?<p>-----<p>BTW, I find these streams to be pedagogically interesting. Unlike, say, Haskell lists, the thunks in the data structure can be stored explicitly. It's a nice exercise in the actual implementation of a lazy sequence.

am i the only one here who finds the whole "omfg, this is <i>so</i> amazing!!1" tone of this article mildly offensive? i mean, it's not like he has a self-levitating hamster or anything.

Sounds like generators in python, but implemented in javascript. The head element and tail function was a interesting way to view them.<p>Generators a great way to do processing on a lot of data, in a very list-y way, without having to read all the data into memory first. It allows you to start seeing results straight away.

The advantage of "streams" is that it can be used to elegantly define infinite sequences recursively.<p>For example, you can define powers of two as:<p><pre><code> var powersOf2 = new Stream(1, function() { return powersOf2.scale(2); }); </code></pre> (untested) which means that the sequence of powers of 2 begins with 1 follows with the the same sequence scaled by 2!<p>The implementation in the linked article however seems to be flawed, since it doesn't store the node's tail after computing it. This means that to get each element of the sequence you have to compute all the elements before it (even though you already did in order to get to that element).<p>This works not only for numbers but for other infinite sequences such as expressions in a language. I recommend the book Higher Order Perl if you're interested in this.

Check out node-lazy, it does something similar and more nicely: <a href="https://github.com/pkrumins/node-lazy" rel="nofollow">https://github.com/pkrumins/node-lazy</a><p><pre><code> var Lazy = require('lazy'); Lazy.range('1..') // infinite list of integers (see readme) .filter(function (n) { return n % 5 == 0 }) .take(5) .join(function (result) { console.log(result) }) ; </code></pre> Output:<p><pre><code> [ 5, 10, 15, 20, 25 ]</code></pre>

It'd be great if the websites for JavaScript libraries like this one would actually include the library in the page, so that while I'm reading about it, I can open up the JavaScript debugging console and actually try out the library as I'm reading about it.

You can create lazy enumerables with linq.js as well.<p><a href="http://neue.cc/reference.htm" rel="nofollow">http://neue.cc/reference.htm</a><p><pre><code> Enumerable .Range(0, 1000000000000) .Take(10) .ToArray() </code></pre> Output:<p><pre><code> 0,1,2,3,4,5,6,7,8,9</code></pre>

This is nothing like a real byte stream, and is very misleading. The name implies that it's for data i/o, when in reality it's for mathematics.

Maybe I don't get it, but what is the practical use-case for this? Infinite arrays of numbers - maybe useful for math or statistical applications, but are people using javascript for this over things like MATLAB, R, etc?<p>I can't think of anywhere to use this in the context of building applications and I certainly don't feel like it "completely changed the way I think about programming" at all.<p>Edit: So it looks like you can put more than just numbers into the streams -- still what does this do that Underscore.js doesn't?

Here's the Python equivalent, for what it's worth: <a href="http://www.trinhhaianh.com/stream.py/" rel="nofollow">http://www.trinhhaianh.com/stream.py/</a><p>I've been playing around with this in my current project, yet I'm not convinced this is always such a great model. It requires a lot of mental work to remember what types you are operating on at each state in the pipeline. That being said, I'm a big fan of LINQ, and I'd like to see this model applied more frequently to DB queries.

For a high-powered version of this written in CoffeeScript which works over arrays and lazy sequences (streams) look here <a href="https://github.com/swannodette/fun.coffee/blob/master/src/fun.coffee" rel="nofollow">https://github.com/swannodette/fun.coffee/blob/master/src/fu...</a><p>ClojureScript of course takes it even further.

"a new data structure" ... really?

For anyone who wasn't paying attention, according to Brendan's slides for TXJS[1][2], TC39 approved iterators and generators for ES, so soon <i>all</i> modern browsers will support them. (Spidermonkey has quietly supported them since Firefox 3—that is, summer 2008.)<p>1. <a href="http://brendaneich.com/2011/08/my-txjs-talk-twitter-remix/" rel="nofollow">http://brendaneich.com/2011/08/my-txjs-talk-twitter-remix/</a><p>2. <a href="http://news.ycombinator.com/item?id=2982256" rel="nofollow">http://news.ycombinator.com/item?id=2982256</a>

I'm used to calling these "Generators", as in a "Generator Function". We could use Forth's defining words to build a pretty succinct version of this type of functionality. Generators will be called by executing their address ("xt" in Forth parlance), and will leave a flag on the stack indicating whether the generator had another element available. If this flag is true, the element produced will be underneath.<p>To start with, let's create a word called "count" which produces a generator which will count from 1 to infinity:<p><pre><code> : count ( -- 'gen ) noname create 0 , latestxt does&#62; ( -- value? flag ) dup @ 1+ dup &#62;r swap ! r&#62; true ; </code></pre> Then we build another generator which slices the first N elements of another generator, so that we can do things without going into an infinite loop:<p><pre><code> : slice ( 'gen count -- 'gen ) noname create , , latestxt does&#62; ( -- value? flag ) dup @ 0= if drop false exit then dup dup @ 1- swap ! cell+ @ execute ; </code></pre> Finally, let's implement "map", so that we can apply a word to every element produced by the generator:<p><pre><code> : map ( 'gen 'func -- ) 2&#62;r begin i' execute if i execute else 2rdrop exit then again ; </code></pre> Now the Forth stack gives us a nice, fluent way of chaining these building blocks together. Let's start with that infinite counter, slice off the first 10 elements and print them out by mapping them to the number printing word ".":<p><pre><code> count 10 slice ' . map </code></pre> And we get:<p><pre><code> 1 2 3 4 5 6 7 8 9 10 </code></pre> Spiffy, hunh?

tl;dr; = It's an array with a few utility methods, but can also hold 'infinite' things like "all the positive even numbers".<p>Not terribly useful imho and certainly doesn't live up to the hyped intro.

Hope this doesn't seem like threadjacking, but your README looked like a fun exercise, so I decided to avoid looking at the code and re-implement in Coffeescript. I wound up memoizing while I was at it.<p><a href="https://github.com/MichaelBlume/coffeestream" rel="nofollow">https://github.com/MichaelBlume/coffeestream</a>

Just curious -- and be honest -- the snippet at the end they ask you to decipher (and they say "Most programmers find it hard to understand unless they have a functional programming background, so don't feel bad if you don't get it immediately") did you have trouble with it?<p>For me, I thought I had the answer after maybe 30 seconds of reading it, but decided to "check my work" by walking through it again, and only then, a moment later, did I realize what it was actually doing with that recursive call.<p>I'm just curious about their "most programmers without a functional background" remark. So: did ya get it?

Here are some key aspects of Streams:<p>- Streams are an alternative to generators (yield)<p>- Streams can be used to traverse recursive structures<p>- Streams implementation only require lambda functions with proper closure<p>- Node.js continuation passing style is a form of Stream<p>I've implemented streams using continuations. You can also see a bunch of related functions such as: print, map, filter, reduce, range, enumerate, comprehension, traverse, empty, cons, zip, xrange, generator2stream.<p><a href="http://blog.vjeux.com/2011/javascript/stream-lazy-iteration-with-continuation.html" rel="nofollow">http://blog.vjeux.com/2011/javascript/stream-lazy-iteration-...</a><p>Tell me if you have any remark :)

I enjoy using the wonderful lazy list implementation here: <a href="https://github.com/pkrumins/node-lazy" rel="nofollow">https://github.com/pkrumins/node-lazy</a> IMHO, it has abetter interface than streamjs

Embracing functional programming in Javascript is trending. All these dollars, underscores and streams will make for an interesting benchmark against ClojureScript as it matures.<p>In ClojureScript, streams or lazy lists are an integral language feature. Standalone javascript libraries can not provide a functional framework of the same calibre. You would naturally end up wanting to, e.g., run an underscore.js chain on a stream and that would require more work.

The guy just finished reading that chapter of SICP? ^_^

I've written an article that talks about the same beast: <a href="http://blog.vjeux.com/2011/javascript/stream-lazy-iteration-with-continuation.html" rel="nofollow">http://blog.vjeux.com/2011/javascript/stream-lazy-iteration-...</a><p>However, I use Javascript functions as base instead of a Stream constructor.

I am sorry but I found the tone of the article a bit offensive. You are not coming out as playful or funny, but as arrogant. Drop the attitude mate, try the guitar if you want to be a rockstar. Apart from that, this is a nice example of how versatile the language can be.

I couldn't get past the 2nd paragraph. the guy's infomercial salesman writing style was hurting my head.

On the surface this looks like a port of Scala's Streams object to Javascript. Any difference?

Seems like they (whoever they is) should add this to Javascript by default.

So how is this different from an array?