Hello Everyone. I'm working on improving Kafka Connect. Kafka connect lacks a lot of data visibility, making it difficult to use and operate as a backbone of data pipelines.<p>We're trying to create a kafka connect for modern data.<p>Right now librarian only supports "Snapshoting", but during winter break I'm going to start hacking on the Streaming replication component.<p>This first version of librarian can snapshot postgres tables and save them as parquet. Although duckdb offers this feature too, librarian provides enhanced data observability through its snapshot "catalog".<p>The catalog provides an inventory of the snapshot including duration, source counts and target counts.<p>------<p>Do you use Kafka connect regularly? What do you use it for? What would you change about it? What works with kafka connect? What's challenging?<p>Thank you all,<p>Hopefully in a couple weeks I have a more mature product to show!