The thing folks don't mention regarding AWS is the inherent competitive advantage their micro-startups have. We focus on AWS launching managed ElasticSearch or managed Kafka, and talk about them (legally) using open source contributions to make money, but I think those are minor compared to things like this.<p>What AWS has is a culture and institutional knowledge on how to launch new products that take foundational AWS services (S3, Lambda, EC2, DDB, etc.) and glues (!) them together better than what a competing non-AWS company can do. This is a bold claim (since AWS launches some very crappy products), but imagine being able to use AWS infrastructure at cost, having internal knowledge on how to best optimize that infrastructure and access to the engineers that own those services while you build abstractions and better user experiences on top of them.<p>I don't know how cos that compete in any related space can survive. When AWS is willing to throw whatever against a wall (launching 50+ services a year) to see what sticks, sooner or later they're going to land in your space.<p>Become more locked into AWS's foundational services -> these abstractions on top of them start to make more sense in engineering complexity / delivery time / possible cost dimensions -> Use more of these -> Become more locked into AWS's foundational services.<p>This feels very different from Azure or GCP.
I am a big fan of AWS and am happily running our entire tech stack with their services for a very reasonable price. That said, Glue is an absolute dumpster fire of a product. My team and I have wasted countless hours trying to wrangle a DynamoDB -> Glue -> Athena -> Quicksight pipeline and Glue refused to cooperate (we ended up building our own DDB to SQL pipeline after finally giving up on Glue). Hopefully this will increase the usability of the Glue product and actually enable out of the box ETL.
Glue is an absolute horrendous mishmash mess that seemed to suffer from a serious lack of investment or vision. The managed spark component is a good product buried under an all-round terrible developer and console UX, and the data catalog/schema crawling is really useful.<p>But I’m glad the lack of investment is turning around with this, the recently released Glue Studio and the fantastic “glue 2 fast startup” job types.
Running any sort of innovative data infrastructure startup (whether data prep, database, data pipeline, etc.) is now an exercise in futility. The big three cloud providers <i>will</i> embrace your innovation, extend your product, and extinguish your business.<p>Given the market power of cloud providers, every infrastructure innovation now a "sustaining innovation" in Christensen's terminology.<p>The key to success seems to be building a product for a niche the cloud providers think is too small, and then either maximizing your value within that niche so that if your market grows large enough for a cloud provider like AWS to come after you, you can pivot to providing customizations to your highest margin customers. MongoDB is a good example of this.<p>On the other hand, none of the major cloud providers seem capable of moving up the value chain to the application level, so if I were starting a company today I would focus on leveraging my infrastructure-level innovation to create a vertical opportunity in a high margin market instead of seeking to build a horizontal platform (IoT platform for example).
Don't buy the copy, buy the original: <a href="https://www.trifacta.com/" rel="nofollow">https://www.trifacta.com/</a>
I'm glad to see a competitor to Trifacta/Google Cloud Dataprep. My company relies on it heavily, but we constantly run into bugs, UI glitches, and crashes that can sometimes block people for hours or days. It's the sort of software that you hate to use, but the benefits are too good to ignore.<p>The benefit to visual ETL is that non-engineers can do a lot of basic data engineering. We tie this into our more complex code-based ETL pipelines. It was a game-changer for us and helps us get a lot more done.
These GUI driven/Visual ETL tools certainly have their place but are firmly at one end of the ease of use vs engineering discipline based ETL continuum.<p>As other posters commented Visual ETL often suffer from source control or limited extension ability but do provide the rapid development environment that users (generally more business oriented) seek. They also tend to trivialize the value of experience/discipline - for example I go to an accountant for my tax because they apply learned-experience relating to tax that I do not have (even though the math is easy) whereas in data engineering seemingly simple tasks such as correctly applying data typing to money or dealing with timezones seems to be glossed over in the pursuit of DIY - and wondering why your money columns don't reconcile or you lose data in failure scenarios.<p>At the other end of the continuum large teams writing bespoke ETL code for every job does not scale well for many reasons (<a href="https://reorchestrate.com/posts/code-doesnt-scale-for-etl/" rel="nofollow">https://reorchestrate.com/posts/code-doesnt-scale-for-etl/</a>). I think the positive reaction to ideas like Data Mesh comes from the failures of these large, centralized teams which coincided with the Hadoop era.<p>Our solution has been to develop an open source (MIT) declarative framework (<a href="https://arc.tripl.ai/" rel="nofollow">https://arc.tripl.ai/</a>) that allows configuration driven ETL - mostly developed via a Jupyter Notebook environment (to allow rapid development and appeal to a larger audience) - whilst making most of the difficult tasks mentioned above easier. This has been in development for a few years now and continues to evolve. We value your feedback.
looks pretty basic limited copy-cat of trifacta/tableau prep/alteryx.<p>this tool requires ready mostly clean-ish data to work with.
but the #1 problem in data engineering is lack of such data
Visual ETL is not as good of an idea as it seems at first. You end up putting a ton of business logic into the menus of these tools, and it’s not version controlled, and it’s not searchable. You’re better off doing this kind of work in SQL.
Having used GUI ETL tools for years (SSIS, Informatca, Talend, Appworx) and now using Airflow(Prefect is an excellent alternative btw), I hope to never go back. Great to see Glue improving and for the industry’s sake I hope it doesn’t catch on. Most ETL should be treated as code. As code, ETLs are easier to write, maintain, and manage complexity.
Have to say as slick as stuff like this looks I do find myself gravtatiing towards ETL in code. (It feels easier to read/understand)<p>How would you change control something like this?
Are there any alternatives to this style of application? I was going to try to make something similar to this for my team to use that would be able to give them access to map columns and simple transforms, then push the resulting flow to me to move it into an airflow dag.<p>I would really like a visual editor I can adjust with Regex functions and mappings that I can self host and iterate on.
<a href="https://conexus.com/" rel="nofollow">https://conexus.com/</a> should get more love. Based off of <a href="https://www.categoricaldata.net/" rel="nofollow">https://www.categoricaldata.net/</a> . Ensures that complex transforms are provably correct.
Haven't used this yet, but this looks like a really good user experience from their demo video. Haven't used competitors like Alteryx myself, but just having this integrate so well into the AWS ecosystem makes this seem really useful.