TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Show HN: ConTube – A Scalable Data Connect Framework for Pulsar/Kafka Ecosystems

1 pointsby aaronrobertover 1 year ago
Hi everyone,<p>I&#x27;m excited to share a project I&#x27;ve been working on, called ConTube. It&#x27;s a scalable data connector framework that aims to simplify and streamline the process of data migration between diverse systems. The key idea behind ConTube is to provide a universal set of data connection protocols, which can work with both Pulsar Connectors and Kafka Connectors, leveraging the rich data connection ecosystem of both systems.<p>For a quick overview, please check this example: <a href="https:&#x2F;&#x2F;github.com&#x2F;RobertIndie&#x2F;contube#example">https:&#x2F;&#x2F;github.com&#x2F;RobertIndie&#x2F;contube#example</a><p>One of the standout features of ConTube is its lightweight runtime. Unlike the Kafka and Pulsar Connect Frameworks, ConTube allows for running multiple tubes within a single JVM process or a single Docker container. This eliminates the need for running separate MQ clusters, which can be a game-changer in terms of resource utilization and operational complexity.<p>Moreover, ConTube provides a universal data format, capable of representing data from virtually any system. This feature is still under development, but the goal is to further enhance the framework&#x27;s versatility and adaptability.<p>I want to stress that ConTube is still in the PoC stage. I&#x27;m hoping to get more invaluable feedbacks and support from the community. I would greatly appreciate any feedback, suggestions, or insights you might have. If you&#x27;re interested in contributing to the development of ConTube, I would be more than happy to welcome you to the project.<p>Please bear in mind that while I&#x27;m proud of what ConTube has accomplished so far, I&#x27;m fully aware that there&#x27;s still a long way to go. I&#x27;m committed to improving and expanding the framework, and I&#x27;m excited to see where our collective efforts can take it.<p>Here&#x27;s the link to the [ConTube repository](<a href="https:&#x2F;&#x2F;github.com&#x2F;RobertIndie&#x2F;contube">https:&#x2F;&#x2F;github.com&#x2F;RobertIndie&#x2F;contube</a>) for you to explore. I&#x27;m looking forward to hearing your thoughts and ideas.<p>Thank you for your time and consideration.

no comments

no comments