Parallel streaming in Haskell: Part 1 – Fast, efficient, and fun

I enjoy programming in conduits but somehow never come across a big enough problem to really make use of them. They encapsulate a whole class of problems in a very comprehensible and modular way. The higher order conduits like `scanlC` and friends also enable more complicated ideas than just the usual `map` and `reduce` variants you usually see. It&#x27;s quite straightforward to model all sorts of chat-&#x2F;websocket-&#x2F;trading-bots as conduits for example, I wrote a blog post about it at <a href="https:&#x2F;&#x2F;wjwh.eu&#x2F;posts&#x2F;2022-05-06-haskell-conduit-bots.html" rel="nofollow">https:&#x2F;&#x2F;wjwh.eu&#x2F;posts&#x2F;2022-05-06-haskell-conduit-bots.html</a>.<p>Btw @rkrzr I hope you don&#x27;t mind me asking but &quot;niet geschoten is altijd mis&quot;: are you looking for freelance assistance at the moment at all? I&#x27;ve been consistently impressed by the engineering work done at Channable, for example the Aho-Corasick work and the stuff relating to compact regions, but am not able to commit to fulltime work at the moment. Lately I&#x27;ve been mostly working with Ruby and DBA&#x2F;db performance stuff, but Haskell is the first language I got properly good at and it will always have a warm place in my heart. I&#x27;m pretty experienced at the intersection between backend dev and DBA&#x2F;devops type problems and if you are experiencing any problems in that area I would love to help out there. Looking at the channable github page, I just realized I even made made some PRs for Icepeak way back in 2020. :)

Nice to see more articles on data flow techniques. They deserve way more attention as a great whole program model for all sorts of problem domains.

Do this streams have any notion of back pressure? I can&#x27;t find info on that.<p>For real world distributed systems back pressure is very important. Otherwise you could DoS yourself by some cascading overload.

Fantastic presentation of this side of FP&#x2F;Haskell I never got to play with. Many thanks.

There are conduit actions (like dedup, sort) that require all the previous action&#x27;s output to be ready and available before they can start producing any output. All the discussion of GC-friendliness of caches makes me think that there aren&#x27;t any conduits that have more data than can fit in memory on a single machine.<p>If you had conduits that were too big to fit in memory at once, would you (channable) stream them to local disk (either explicitly or just using virtual memory)? Or would you be able to distribute work between multiple machines with a cluster-aware Conduit type?<p>Your scheduler could split the job up into multiple machines and run the same Conduit pipeline on all the machines, and only the conduit steps that need to communicate with each other would do so.<p>Separate question, do you have any actions that produce more output than their input? I could imagine some customers might find it useful to generate the cartesian product of two inputs, or the power set of one input.

Cool! Is this the same thing as streams in SICP?