TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Launch HN: Hatchet (YC W24) – Open-source task queue, now with a cloud version

245 pointsby abelanger11 months ago
Hey HN - this is Alexander and Gabe from Hatchet (<a href="https:&#x2F;&#x2F;hatchet.run">https:&#x2F;&#x2F;hatchet.run</a>). We’re building a modern task queue as an alternative to tools like Celery for Python and BullMQ for Node. Our open-source repo is at <a href="https:&#x2F;&#x2F;github.com&#x2F;hatchet-dev&#x2F;hatchet">https:&#x2F;&#x2F;github.com&#x2F;hatchet-dev&#x2F;hatchet</a> and is 100% MIT licensed.<p>When we did a Show HN a few months ago (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39643136">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39643136</a>), our cloud version was invite-only and we were focused on our open-source offering.<p>Today we’re launching our self-serve cloud so that anyone can get started creating tasks on our platform - you can get started at <a href="https:&#x2F;&#x2F;cloud.onhatchet.run" rel="nofollow">https:&#x2F;&#x2F;cloud.onhatchet.run</a>, or you can use these credentials to access a demo (should be prefilled):<p><pre><code> URL: https:&#x2F;&#x2F;demo.hatchet-tools.com Email: hacker@news.ycombinator.com Password: HatchetDemo123! </code></pre> People are currently using Hatchet for a bunch of use-cases: orchestrating RAG pipelines, queueing up user notifications, building agentic LLM workflows, or scheduling image generation tasks on GPUs.<p>We built this out of frustration with existing tools and a conviction that PostgreSQL is the right choice for a task queue. Beyond the fact that many developers are already using Postgres in their stack, which makes it easier to self-host Hatchet, it’s also easier to model higher-order concepts in Postgres, like chains of tasks (which we call workflows). In our system, the acknowledgement of the task, the task result, and the updates to higher-order models are done as part of the same Postgres transaction, which significantly reduces the risk of data loss&#x2F;race conditions when compared with other task queues (which usually pass acknowledgements through a broker, storing the task results elsewhere, and only then figuring out the next task in the chain).<p>We also became increasingly frustrated with tools like Celery and the challenges it introduces when using a modern Python stack (&gt; 3.5). We wrote up a list of these frustrations here: <a href="https:&#x2F;&#x2F;docs.hatchet.run&#x2F;blog&#x2F;problems-with-celery">https:&#x2F;&#x2F;docs.hatchet.run&#x2F;blog&#x2F;problems-with-celery</a>.<p>Since our Show HN, we’ve (partially or completely) addressed the most common pieces of feedback from the post, which we’ll outline here:<p>1. The most common ask was built-in support for fanout workflows — one task which triggers an arbitrary number of child tasks to run in parallel. We previously only had support for DAG executions. We generalized this concept and launched child workflows (<a href="https:&#x2F;&#x2F;docs.hatchet.run&#x2F;home&#x2F;features&#x2F;child-workflows">https:&#x2F;&#x2F;docs.hatchet.run&#x2F;home&#x2F;features&#x2F;child-workflows</a>). This is the first step towards a developer-friendly model of durable execution.<p>2. Support for HTTP-based triggers — we’ve built out support for webhook workers (<a href="https:&#x2F;&#x2F;docs.hatchet.run&#x2F;home&#x2F;features&#x2F;webhooks">https:&#x2F;&#x2F;docs.hatchet.run&#x2F;home&#x2F;features&#x2F;webhooks</a>), which allow you to trigger any workflow over an HTTP webhook. This is particularly useful for apps on Vercel, who are dealing with timeout limits of 60s, 300s, or 900s (depending on your tier).<p>3. Our RabbitMQ dependency — while we haven’t gotten rid of this completely, we’ve recently launched hatchet-lite (<a href="https:&#x2F;&#x2F;docs.hatchet.run&#x2F;self-hosting&#x2F;hatchet-lite">https:&#x2F;&#x2F;docs.hatchet.run&#x2F;self-hosting&#x2F;hatchet-lite</a>), which allows you to run the various Hatchet components in a single Docker image that bundles RabbitMQ along with a migration process, admin CLI, our REST API, and our gRPC engine. Hopefully the lite was a giveaway, but this is meant for local development and low-volume processing, on the order of hundreds per minute.<p>We’ve also launched more features, like support for global rate limiting, steps which only run on workflow failure, and custom event streaming.<p>We’ll be here the whole day for questions and feedback, and look forward to hearing your thoughts!

28 comments

krick11 months ago
Can somebody explain why would I use it instead a simple Redis&#x2F;SQS&#x2F;Postgres queue implemented in 50 LOC (+ some grafana panel for monitoring) (which is pretty much mandatory even for a wrapper of this or any other service)? I&#x27;m not trying to mock it, it&#x27;s a serious question. What is implied by &quot;task queue&quot; that makes it worth bothering to use a dedicated service?
评论 #40819903 未加载
评论 #40817933 未加载
评论 #40818217 未加载
评论 #40817478 未加载
评论 #40817480 未加载
dalberto11 months ago
I&#x27;m super interested in a Postgres-only task queue, but I&#x27;m still unclear from your post whether the only broker dependency is PostgreSQL. You mention working towards getting rid of the RabbitMQ dependency but the existence of RabbitMQ in your stack is dissonant with the statement &#x27;a conviction that PostgreSQL is the right choice for a task queue&#x27;. In my mind, if you are using Postgres as a queue, I&#x27;m not sure why you&#x27;d also have RabbitMQ.
评论 #40811580 未加载
fangpenlin11 months ago
Hatchet looks pretty awesome. I was thinking about using it to replace my Celery worker. However, the problem is that I can only use the gRPC client to create a task (correct me if I am wrong). What I want is to be able to commit a bunch of database rows altogether with the background task itself directly. The benefit of doing so with a PostgreSQL database is that all the rows will be in the same transaction. With traditional background worker solutions, you will run into two problems:<p>1. Commit changes in the db first: if you fail to enqueue the task, there will be data rows hanging in the db but no task to process them<p>2. Push the task first: the task may kick start too early, and the DB transaction is not committed yet, it cannot find the rows still in transaction. You will need to retry failure<p>We also looked at Celery and hope it can provide a similar offer, but the issue seems open for years:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;celery&#x2F;celery&#x2F;issues&#x2F;5149">https:&#x2F;&#x2F;github.com&#x2F;celery&#x2F;celery&#x2F;issues&#x2F;5149</a><p>With the needs, I build a simple Python library on top of SQLAlchemy:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;LaunchPlatform&#x2F;bq">https:&#x2F;&#x2F;github.com&#x2F;LaunchPlatform&#x2F;bq</a><p>It would be super cool if Hatchet also supports native SQL inserts with ORM frameworks. Without the ability to commit tasks with all other data rows, I think it&#x27;s missing out a bit of the benefit of using a database as the worker queue backend.
评论 #40812674 未加载
numlocked11 months ago
Being MIT licensed, does that mean that another company could also offer this as a hosted solution? Did you think about encumbering with a license that allowed commercial use, but prohibited resale?<p>Also, somewhat related, years ago I wrote a very small framework for fan-out of Django-based tasks in Celery. We have been running it in production for years. It doesn&#x27;t have adoption beyond our company, but I think there are some good ideas in it. Feel free to take a look if it&#x27;s of interest! <a href="https:&#x2F;&#x2F;github.com&#x2F;groveco&#x2F;django-sprinklers">https:&#x2F;&#x2F;github.com&#x2F;groveco&#x2F;django-sprinklers</a>
评论 #40815012 未加载
评论 #40814767 未加载
评论 #40814314 未加载
评论 #40813641 未加载
acaloiar11 months ago
I love seeing commercial activity around using Postgres as a queue. Last year I wrote a post titled &quot;Choose Postgres queue technology&quot; that spent quite a bit of time on the front page here. I don&#x27;t think it&#x27;s likely that my post actually sparked new development in this area, but for the people who were already using Postgres queues in their applications, I hope it made them feel more comfortable talking about it in public. And I&#x27;ve seen a notable increase in public discussions around the idea, and they&#x27;re not all met with derision. There&#x27;s long been a dogma around Postgres and relational databases being the wrong tool for the job, and indeed they are not perfect, but neither is adding Redis or RabbitMQ to our software stacks simply to support queue use cases. Kudos to the Hatchet team! I hope you all find success.
评论 #40814594 未加载
评论 #40814248 未加载
评论 #40813580 未加载
评论 #40813240 未加载
评论 #40816599 未加载
mads_quist11 months ago
Awesome! Reducing moving parts is always a great thing!For 99.9% of teams this is a great alternative to rely only on the database the team&#x27;s already using. For those teams that use MongoDB, I created something similar (and simpler of course): <a href="https:&#x2F;&#x2F;allquiet.app&#x2F;open-source&#x2F;mongo-queueing" rel="nofollow">https:&#x2F;&#x2F;allquiet.app&#x2F;open-source&#x2F;mongo-queueing</a> The package is C#, but the idea could be adapted to practically any language that has a MongoDB driver.
andrewstuart11 months ago
I&#x27;ve built several message queues over the years.<p>I hated the configuration and management complexity of RabbitMQ and Celery and pretty much everything else.<p>My ultimate goal was to build a message queue that was extremely fast and required absolutely zero config and was HTTP based thus has no requirement for any specific client.<p>I developed one in Python that was pretty complete but slow, then developing a prototype in Rust that was extremely fast but incomplete.<p>The latest is sasquatch. Its written in golang, uses sqlite for the db and behaves in a very similar way to Amazon SQS in that connections are HTTP and it uses long polling to wait for messages.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;crowdwave&#x2F;sasquatch">https:&#x2F;&#x2F;github.com&#x2F;crowdwave&#x2F;sasquatch</a><p>Its only in the very early stages of development at this stage and likely isn&#x27;t even compiling but most of the code is in place. I&#x27;m hoping to get around to next phase of development soon.<p>I just love the idea of a message queue that is a single static binary and when you run it, you have a fully functioning message queue with nothing more to do - not even fiddling with Postgres.<p>Absolute zero config, not minutes, hours or days of futzing with configs and blogs and tutorials.
nickzelei11 months ago
Interesting and congrats on the launch!<p>I am definitely a fan of all things postgres and it&#x27;s great to see another solution that uses it.<p>My main thing is the RabbitMQ dependency (that seems to be a topic of interest in this thread). Getting rid of that and just depending on PG seems like the main path forward that would increase adoption. Right now I&#x27;d be considering something like this over using a tool like Rabbit (if I were making that consideration.)<p>You also compare yourself against Celery and BullMQ, but there is also talk in the readme around durable execution. That to me puts you in the realm of Temporal. How would you say you compare&#x2F;compete with Temporal? Are you looking to compete with them?<p>EDIT: I also understand that Rabbit comes with certain things (or rather, lacks certain things) that you are building ontop of, which is cool. It&#x27;s easy to say: why are you using rabbit?? but if it&#x27;s allowing you to function like it with new additions&#x2F;features, seems like a good thing!
评论 #40812355 未加载
cyral11 months ago
How does this compare to Temporal or Inngest? I&#x27;ve been investigating them and the durable execution pattern recently and would like to implement one soon.
评论 #40813180 未加载
评论 #40813704 未加载
评论 #40812515 未加载
评论 #40813325 未加载
评论 #40812987 未加载
grogenaut11 months ago
Shameless plug since I never get to do those: <a href="https:&#x2F;&#x2F;github.com&#x2F;gaffo&#x2F;jorb">https:&#x2F;&#x2F;github.com&#x2F;gaffo&#x2F;jorb</a><p>There&#x27;s many great distributed job runners out there. I&#x27;ve never found one for go that lets me have the features without running 7 processes and message queues sprawled over hosts and docker containers.<p>jorb is just a framework to slap into a go script when you want to fire a lot of work at your computer and let it run it to completion.<p>I&#x27;ve tried to build this many times and this is the first time I&#x27;ve gotten it to stick.<p>Yes you can do this with core go primitives but I find this abstraction to be a lot better and (eventually) was easier to debug deadlocks.<p>I&#x27;m just putting it here cause it&#x27;s semi related.
barrell11 months ago
I’ve been through a whole journey with distributed tasks queues - from celery, to arq, to recently hatchet. Not only is hatchet the only solution that doesn’t make me want to tear my hair out, but the visibility the product gives you is amazing! Being able to visually explore logs, props, refrigerate specific queues etc has been a game changer,<p>Also, minor thing, but the granularity around rate limiting and queues also feels like quite the luxury. Excited for more here too<p>Cool to see them on the front page, congrats on the launch
评论 #40816057 未加载
didip11 months ago
I am surprised that there&#x27;s still money for this type of OSS SaaS companies.<p>Aren&#x27;t all the money go to AI companies these days (even the unicorns didn&#x27;t do well with their IPOs. E.g. Hashicorp).<p>That said, I love every single addition to the Go community so thumbs up from me.
评论 #40816361 未加载
评论 #40814328 未加载
mind-blight11 months ago
I&#x27;m really curious how you folks compare to something like Apache Airflow. They do a similar durable execution w&#x2F; DAGs on top of postgres and redis. They&#x27;re Python-only (one definite difference). I&#x27;m curious what other comparisons you see<p>ETA: I really like the idea of this being entirely built on Postgres. That makes infrastructure a lot easier to manage
评论 #40814163 未加载
gabev11 months ago
Hey, this is Gabe from zenfetch. Been following you guys for a few months now since your first launch. I definitely resonate with all the problems you&#x27;ve described regarding celery shortcomings &#x2F; other distributed task queues. We&#x27;re on celery right now and have been through the ringer with various workflow platforms. Only reason we haven&#x27;t switched to Hatchet is because we are finally in a stable place, though that might change soon in which case I&#x27;d be very open to jumping ship.<p>I know a lot of folks are going after the AI agent workflow orchestration platform, do you see yourselves progressing there?<p>In my head, Hatchet coupled with BAML (<a href="https:&#x2F;&#x2F;www.boundaryml.com&#x2F;">https:&#x2F;&#x2F;www.boundaryml.com&#x2F;</a>) could be an incredible combination to support these AI agents. Congrats on the launch
评论 #40812246 未加载
wenbin11 months ago
Looks awesome.<p>We&#x27;ve been using Celery at ListenNotes.com since 2017. I agree that observability of Celery tasks is not great.
subham-padhi11 months ago
How is this technically different from what solid queue is doing on rails ?<p><a href="https:&#x2F;&#x2F;github.com&#x2F;rails&#x2F;solid_queue">https:&#x2F;&#x2F;github.com&#x2F;rails&#x2F;solid_queue</a><p>Just trying to understand. I do get that hatchet would be language agnostic, SDK API kind of a solution.
cedws11 months ago
What happened to the Terraform management tool? Pivot?
评论 #40811817 未加载
n00bskoolbus11 months ago
This looks really awesome! We were just discussing at work how we&#x27;re having a hard time finding a framework for a task queue that supports dependant tasks and has support for Python &amp; TS. I suppose writing that out it does feel like a pretty specific requirement. I&#x27;m glad to see this pop up though, feels very relevant to me right now.<p>A question around workflows having just skimmed your docs. Is it possible to define a workflow that has steps in Python and a TS app?
评论 #40815827 未加载
ocolegro11 months ago
We&#x27;ve been using hatchet for cloud deployments and have really enjoyed the reliable execution &#x2F; observability, congrats on the launch.
klysm11 months ago
Looks cool, but I’m still team everything-in-Postgres
评论 #40812966 未加载
评论 #40813881 未加载
评论 #40812183 未加载
soohoonchoi11 months ago
we use hatchet to orchestrate our long running backend jobs. it provided us with scalability, reliability, and observability into our tasks with a couple lines of code.
michaelmarkell11 months ago
can you run the whole task as a postgres transaction? like if i want to make an idempotent job by only updating some status to &quot;complete&quot; once the job finishes.
评论 #40811557 未加载
评论 #40811529 未加载
评论 #40813391 未加载
h0h0h011 months ago
I was going to reach for dagster to run some DBT and data load scripts. No orchestration as of yet-why would I use hatchet instead of dagster?
distributedsean11 months ago
Nice, looks really good. High time a decent task queue came along that is usable with the Node ecosystem.
jusonchan8111 months ago
What are some real world use cases you see customers using this for?
评论 #40814369 未加载
thevivekshukla11 months ago
Seems interesting, what are the plans on Rust SDK?
评论 #40814449 未加载
stevefan199911 months ago
Is .NET&#x2F;C# support on the roadmap?
smallshen11 months ago
I&#x27;m wondering what is the difference from <a href="https:&#x2F;&#x2F;docs.urlinks.io&#x2F;gateway-chain&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.urlinks.io&#x2F;gateway-chain&#x2F;</a> . There are a lot similar concepts, like very similar. Hatchet feels like same product but with money from VCs.
评论 #40816359 未加载