This article is somewhat puzzling for me. On one hand, the OP clearly knows Clojure very well. The disadvantages of laziness are real and well described.<p>On the other hand, though, this sounds like a theoretical/academic article to me. I've been using Clojure for 15 years now, 8 of those developing and maintaining a large complex SaaS app. I've also used Clojure for data science, working with large datasets. The disadvantages described in the article bothered me in the first 2 years or so, and never afterwards.<p>Laziness does not bother me, because I very rarely pass lazy sequences around. The key here is to use transducers: that lets you write composable and reusable transformations that do not care about the kind of sequence they work with. Using transducers also forces you to explicitly realize the entire resulting sequence (note that this does not imply that you will realize the entire <i>source</i> sequence!), thus limiting the scope of lazy sequences and avoiding a whole set of potential pitfalls (with dynamic binding, for example), and providing fantastic performance.<p>I do like laziness, because when I need it, it's there. And when you need it, you are <i>really</i> happy that it's there.<p>In other words, it's something I don't think much about anymore, and it doesn't inconvenience me in any noticeable way. That's why I find the article puzzling.
As I posted on Reddit:<p>It might also be good to mention Injest<p><a href="https://github.com/johnmn3/injest">https://github.com/johnmn3/injest</a><p>Which makes transducers more ergonomic to use if you are like me and use threading macros everywhere<p>Would be curious to hear how others feel about it
It’s too bad that transducers were created long after clojure’s inception. Can you always replace a lazy seq with a transducer? Could the language theoretically be redesigned to replace all default usages of lazy seqs with transducers, even if it were a major breaking change? And have lazy operations be very explicit?
Not sure if the "transducer" approach suggested as a workaround makes your life easier or further adds to the mental overhead. See <a href="https://www.astrecipes.net/blog/2016/11/24/transducers-how-to/" rel="nofollow noreferrer">https://www.astrecipes.net/blog/2016/11/24/transducers-how-t...</a> for some example transducers.
I've been circling around lisp for a couple of years. I'm starting in a month, I'll spend several hours a day. I still don't know what language I want to learn.<p>I was drawn to Clojure because it looked like a lisp for getting stuff done. But a few things put me off. This article puts me off more. I want to get the semantics down before I have to think about what's going on under the hood.
The main issue is that Clojure compiler doesn't really optimize lazy sequences right ? Most language compilers do this. Rust lazy iterators for example many times exhibit faster performance than for-lops.<p>And clojure also doesn't give an error/warning when lazy sequences aren't finalized.
I've been using the trick with enforcing realization by serializing to strings a few times. Slow, but quite useful in many contexts. However, instead of using `(with-out-str (pr ...`, there's simply`pr-str`, which is easier to remember.<p>I'm typically using it like so:<p><pre><code> (defn realize [v] (doto v pr-str))
(binding [*some* binding]
(realize (f some-nested-lazy-seq)))</code></pre>
F# is similar in that it supports lazy sequences but is mostly eager otherwise, and often handles errors using exceptions. One does have to be careful, but the benefits far outweigh the risks in my experience.
I think the main issue with lazy sequences is understanding and controlling their scope. Transducers, particularly when utilized within an `into` scope, can encapsulate laziness very neatly. Indeed, transducers utilize lazy sequences internally, and the OP shows their clear performance advantage. I think the article would be more effective if it shifted tone to "Clojure laziness best practices" rather than damning the idea wholesale. There be dragons for sure.
TXR Lisp also fails this test:<p><pre><code> 1> (len
(with-stream (s (open-file "/usr/share/dict/words"))
(get-lines s)))
** error reading #<file-stream /usr/share/dict/words b7ad7270>: file closed
** during evaluation of form (len (let ((s (open-file "/usr/share/dict/words")))
(unwind-protect
(get-lines s)
(close-stream s))))
** ... an expansion of (len (with-stream
(s (open-file "/usr/share/dict/words"))
(get-lines s)))
** which is located at expr-1:1
</code></pre>
The built-in solution is that when you create a lazy list which reads lines from a stream, that lazy list takes care of closing the stream when it is done.<p>If the lazy list isn't processed to the end, then the stream semantically leaks; it has to be cleaned up by the garbage collector when the lazy list becomes unreachable.<p>We can see with strace that the stream is closed:<p><pre><code> $ strace txr -p '(flow "/usr/share/dict/words" open-file get-lines len)'
[...]read(3, "d\nwrapper\nwrapper's\nwrappers\nwra"..., 4096) = 4096
read(3, "zigzags\nzilch\nzilch's\nzillion\nzi"..., 4096) = 826
read(3, "", 4096) = 0
close(3) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
write(1, "102305\n", 7102305
) = 7
exit_group(0) = ?
+++ exited with 0 +++
</code></pre>
It is possible to address the error issue with reference counting. Suppose that we define a stream with a reference count, such that it has to be closed that many times before the underlying file descriptor is closed.<p>I programmed a proof of concept of this today. (I ran into a small issue in the language run-time that I fixed; the close-stream function calls the underlying method and then caches the result, preventing the solution from working.)<p><pre><code> (defstruct refcount-close stream-wrap
stream
(count 1)
(:method close (me throw-on-error-p)
(put-line `close called on @me`)
(when (plusp me.count)
(if (zerop (dec me.count))
(close-stream me.stream throw-on-error-p)))))
(flow
(with-stream (s (make-struct-delegate-stream
(new refcount-close
count 2
stream (open-file "/usr/share/dict/words"))))
(get-lines s))
len
prinl)
</code></pre>
With my small fix in stream.c (already merged, going into Version 292), the output is:<p><pre><code> $ ./txr lazy2.tl
close called on #S(refcount-close stream #<file-stream /usr/share/dict/words b7aecee0> count 2)
close called on #S(refcount-close stream #<file-stream /usr/share/dict/words b7aecee0> count 1)
102305
</code></pre>
One close comes from the with-stream macro, the other from the lazy list hitting EOF when its length is being calculated.<p>Without the fix, I don't get the second call; the code works, but the descriptor isn't closed:<p><pre><code> $ txr lazy2.tl
close called on #S(refcount-close stream #<file-stream /usr/share/dict/words b7b70f10> count 2)
102305
</code></pre>
In the former we see the call to close in strace; in the latter we don't.