(i work at airbyte) this was announced last month at our conf to a very positive response (<a href="https://movedata.airbyte.com/">https://movedata.airbyte.com/</a>, keynote)<p>The context is that Airbyte is now (after pivoting 3x during YC <a href="https://airbyte.com/blog/how-we-pivoted-3-times-in-the-1st-month-of-yc">https://airbyte.com/blog/how-we-pivoted-3-times-in-the-1st-m...</a>) the largest/fastest growing open source community (see our github <a href="https://github.com/airbytehq/airbyte">https://github.com/airbytehq/airbyte</a>) of data pipeline connectors[0], so in a sense they have always been free if you are self hosting. But now <i>using them on Airbyte Cloud</i> is going to be free as well aka "we will do your ELT for free no matter the volume as long as our connectors are not GA yet".<p>This is a massive commitment to improve the quality of our connectors, which is also something we have been pushing the industry on: <a href="https://airbyte.com/blog/connector-release-stages">https://airbyte.com/blog/connector-release-stages</a> :<p>Alpha: new, basic docs, works, passes acceptance tests<p>Beta: Alpha + at least 25 active users + >90% sync success rate + snapshot tests + all streams + severe issues handled + security + supports checkpointing + SLA on cloud<p>GA: Beta + >99% sync success rate + more than 50 active users + <24 hours downtime + polished docs + performant<p>It's been going very well; you can see how many connectors we promote to GA each month in our slack (<a href="https://slack.airbyte.io/" rel="nofollow">https://slack.airbyte.io/</a>) and changelogs, and our new lowcode CDK (<a href="https://www.youtube.com/watch?v=i7VSL2bDvmw">https://www.youtube.com/watch?v=i7VSL2bDvmw</a>) is helping new connectors insta-promote to beta.<p>We hope to set the new standard in data integration and this is still only day 1.<p>[0]: good explainer on why companies are moving towards ELT in the first place for the uninitiated <a href="https://airbyte.com/blog/elt-pipeline">https://airbyte.com/blog/elt-pipeline</a>
Oh man, so happy to see this.<p>For me right now, Airbyte is that tool I wish we had at my last startup.<p>We we're pulling data from a lot of weird places (servers in the back of mom and pop vet clinics). This meant writing a lot of one off scripts to populate our databases. We learned the hard way about scheduling, retries, resource monitoring, error reporting etc..<p>Would've loved to have someone else take care of all that for us.<p>Anyway love letter over.<p>I'm currently wondering if I can use this to power some of my web scraping scripts....
Airbyte is a godsend for us. It works really well for most use cases. Unfortunately, we had to write our custom thing for a large table (8 billion rows) and "tricked" Airbyte into thinking it had done the first sync. After that, it continued to sync happily.<p>Hoping that more users will bring more maturity and better solutions to those edge cases.
I have recently inherited the small and unsophisticated data engineering practice at my company and it's been a learning experience. The market for tooling seems to be incredibly frothy right now and I'm almost at a loss to make good selections. Is airbyte a direct competitor to stuff like fivetran or AWS Glue?
Congrats! Shameless plug (Original Author and Founder) - Go High Performance alternative (<a href="http://github.com/cloudquery/cloudquery">http://github.com/cloudquery/cloudquery</a>). no backend, no ui, everything stateless, as code and you can run it anywhere including ECS.
How do you know what the status of a connector is? I just see <a href="https://airbyte.com/connectors">https://airbyte.com/connectors</a> and have no obvious indication of status.