For anyone who has spent significant time with data and ML pipelines, read posts about Dagster. Even if you don't end up using it in your pipeline, it is full of well articulated and extremely valuable lessons about how to manage the problems and complexity of data applications. I can't recommend reading about dagster enough!<p>The software itself has matured significantly in the last year as well - highly recommend taking a look at this if you're building new pipelines or have a need to upgrade existing ones.
After playing around with this tool a bit in the past I can't seem to understand what it's trying to solve. It comes across as a side project that VCs happened to throw money at.