Sail – Unify stream processing, batch processing and compute-intensive workloads

From the announcement “As of now, we have mined 1,580 PySpark tests from the Spark codebase, among which 838 (53.0%) are successful on Sail. We have also mined 2,230 Spark SQL statements or expressions, among which 1,396 (62.6%) can be parsed by Sail”<p>Kinda early to call this a drop in replacement with those numbers no?<p>But, with enough parity this project could be a dream for anybody dealing with spark’s dreadful performance. Kudos to the team

Bit off topic; we are looking for something like this but with a facility for untrusted users to run sandboxed code instead of trusted code. All that I found (but I am relatively new to this field) are hacky and, worse, slow solutions.

It is refreshing to see multiple projects with arrow&#x2F;datafusion trying to bank on existing and user friendly spark&#x27;s API instead of reinventing the API all over again.<p>There is likes of comet and blaze that replace execution backend of spark with datafusion and then you have single process alternatives like sail trying to settle in &quot;not so big data&quot; category.<p>I am watching evolution of projects powered by datafusion and compatible with spark with keen eye. Early days but quite exciting.

This looks interesting, but the docs are really lacking, to the point where it is barely understandable.<p>I see some potential wins on it, such as it being a Rust-based, Spark-compatible and better suited for single processor environments, but they are just not explained or developed enough.

Been on hacker news long enough to take bold claims like this with a few cups of salt but bookmarked so some kind of interest sparked (hah!).

Is this for distributed data processing like spark is?<p>The documentation has nothing to indicate that it is.

&quot;In single-process settings&quot;<p>nobody uses Spark for this outside of unit testing

Note. This is not Laravel Sail - the docker environment.