TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Show HN: Drop-in SQS replacement based on SQLite

656 点作者 memset11 个月前
Hi! I wanted to share an open source API-compatible replacement for SQS. It&#x27;s written in Go, distributes as a single binary, and uses SQLite for underlying storage.<p>I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have.<p>But I didn&#x27;t want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific.<p>So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS&#x27; native library in your language of choice.<p>For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn&#x27;t support SQS.)<p>It is written to be pluggable. The queue implementation uses SQLite, but I&#x27;ve been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot.<p>It is written to be as easy to deploy as possible - a single go binary. I&#x27;m working on adding distributed and autoscale functionality as the next layer.<p>Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future.<p>In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues.<p>I&#x27;d love your feedback!

35 条评论

davidthewatson11 个月前
The whole idea here is superlative.<p>+1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost.<p>I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge.<p>Private cloud matters. This is an enabler. We&#x27;ve done too much already in public cloud where many things don&#x27;t belong.<p>BTLE sensors are perfectly happy talking to my Apple Watch directly with enough debugging.<p>I&#x27;d argue the trip through cloud was not a win and should be corrected in the next generation of tools like this, where mobile is already well-primed for SQLite.
评论 #40838793 未加载
评论 #40842456 未加载
评论 #40839276 未加载
hrisen11 个月前
Keeping questions from scale &amp; benchmarks aside, this is a cool thing for functional&#x2F;unit testing module that uses SQS, instead of dumb mocks.
评论 #40838218 未加载
dav4311 个月前
Tangential but I looked at the codebase - because I like to imagine I can code (I can’t) - and for a laymen, I could follow the code.<p>Makes me think quite positively of go lang and of the dev for designing in such a way. Can understand why teams like it because of easier maintenance<p>Quite elegant from my uneducated pov.
评论 #40842674 未加载
评论 #40841809 未加载
评论 #40841852 未加载
评论 #40842106 未加载
jerrygenser11 个月前
&gt; In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues.<p>are you monetizing this as a separate business from: <a href="https:&#x2F;&#x2F;www.ycombinator.com&#x2F;companies&#x2F;scratch-data">https:&#x2F;&#x2F;www.ycombinator.com&#x2F;companies&#x2F;scratch-data</a>
评论 #40838104 未加载
评论 #40838475 未加载
4RealFreedom11 个月前
The goals are a little different but I think it&#x27;s worth pointing out ElasticMQ. I use it simulate sqs in a docker environment. <a href="https:&#x2F;&#x2F;github.com&#x2F;softwaremill&#x2F;elasticmq">https:&#x2F;&#x2F;github.com&#x2F;softwaremill&#x2F;elasticmq</a>
评论 #40843843 未加载
评论 #40838450 未加载
yimpolo11 个月前
This is super cool! I love projects that aim to create simple self-hostable alternatives to popular services.<p>I assume this would work without much issue with Litestream, though I&#x27;m curious if you&#x27;ve already tried it. This would make a great ephemeral queue system without having to worry about coordinating backend storage.
评论 #40838294 未加载
ahachete11 个月前
Congratulations!<p>I also love writing AWS API-compatible services. That&#x27;s why I did Dyna53 [1] ;P<p>(I know, unrelated, but hopefully funny)<p>[1] <a href="https:&#x2F;&#x2F;dyna53.io" rel="nofollow">https:&#x2F;&#x2F;dyna53.io</a>
评论 #40841784 未加载
评论 #40842460 未加载
评论 #40841541 未加载
philippta11 个月前
One quick suggestion on project structure:<p>Move all the structs from models&#x2F; into the root directory.<p>This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.
评论 #40838771 未加载
评论 #40840503 未加载
PanMan11 个月前
Correct me if I’m wrong but: SQLite sounds like: runs on one server. While that will work most of the time, it won’t work 100% of the time. I don’t know the specifics, but I’m fairly sure if a queue server crashes, SQS will keep working, as stuff is redundant. So while it can work in a best case, this (probably) won’t have the same reliability as SQS has..
评论 #40840947 未加载
评论 #40845056 未加载
评论 #40841867 未加载
stuaxo11 个月前
Good, we need open implementations of all the AWS stuff.<p>I swear they reimplement stuff we have just so there are more places to bill us.
评论 #40838223 未加载
评论 #40839874 未加载
john-tells-all11 个月前
Why not use LocalStack? It has SQS and a <i>lot</i> of AWS services for testing&#x2F;development. Well documented, open source.<p><a href="https:&#x2F;&#x2F;docs.localstack.cloud&#x2F;overview&#x2F;" rel="nofollow">https:&#x2F;&#x2F;docs.localstack.cloud&#x2F;overview&#x2F;</a>
评论 #40840258 未加载
评论 #40840363 未加载
评论 #40840855 未加载
koito1711 个月前
I may be asking a naive question, but what is the rationale behind disabling foreign key support and using them anyway in the database schema? See <a href="https:&#x2F;&#x2F;github.com&#x2F;poundifdef&#x2F;SmoothMQ&#x2F;blob&#x2F;46f8b22&#x2F;queue&#x2F;sqlite&#x2F;sqlite.go#L56-L88">https:&#x2F;&#x2F;github.com&#x2F;poundifdef&#x2F;SmoothMQ&#x2F;blob&#x2F;46f8b22&#x2F;queue&#x2F;sq...</a><p>The &quot;TODO: check for errors&quot; comment, combined with what seems like disabling foreign key constraint checks, makes me a bit hesitant to try this out.
评论 #40840114 未加载
chromanoid11 个月前
Could you elaborate why you didn&#x27;t choose e.g. RabbitMQ? I mean you talk about AMQP and such, it seems to me, that a client-side abstraction would be much more efficient in providing an exit strategy for SQS than creating an SQS &quot;compatible&quot; broker. For example in the Java ecosystem there is <a href="https:&#x2F;&#x2F;smallrye.io&#x2F;smallrye-reactive-messaging&#x2F;latest&#x2F;" rel="nofollow">https:&#x2F;&#x2F;smallrye.io&#x2F;smallrye-reactive-messaging&#x2F;latest&#x2F;</a> that serves a similar purpose.
评论 #40842365 未加载
andrewstuart11 个月前
I wrote sasquatch which is similar. sasquatch is more simple than SQS and queues are virtual (i.e the queue exists because you put a message in it, and is gone when there&#x27;s no messages in it).<p><a href="https:&#x2F;&#x2F;github.com&#x2F;crowdwave&#x2F;sasquatch">https:&#x2F;&#x2F;github.com&#x2F;crowdwave&#x2F;sasquatch</a><p>sasquatch is also a message queue, also written in Golang and also based on sqlite.<p>sasquatch implements behaviour very similar to SQS but does not attempt to be a dropin replacement.<p>sqsquatch is not a complete project though, nor even really a prototype, just early code. Likely it does not compile.<p>HOWEVER - sasquatch is MIT license (versus this project which is AGPL) so you are free to do with it as you choose.<p>sasquatch is a single file of 700 lines so easy to get your head around: <a href="https:&#x2F;&#x2F;raw.githubusercontent.com&#x2F;crowdwave&#x2F;sasquatch&#x2F;main&#x2F;sasquatch.go" rel="nofollow">https:&#x2F;&#x2F;raw.githubusercontent.com&#x2F;crowdwave&#x2F;sasquatch&#x2F;main&#x2F;s...</a><p>Just remember as I say it&#x27;s early code, won&#x27;t even compile yet but functionally should be complete.
mannyv11 个月前
So I assume it does the back-end as well?<p>I never cared to figure out what parts of SQS are clients-side and server side, but - does SmoothMQ support long polling, batch delivery, visibility timeouts, error handling, and - triggers? Or are triggers left to whatever is implementing the queue? Both FiFo and simple queues? Do you have throughput numbers?<p>As an SQS user, a table of SQS features vs SmoothMQ would be handy. If it&#x27;s just an API-compatible front-end then that would be good to know. But if it does more that would also be good to know.<p>The reason you&#x27;d use this is because there are lots of clients who still want on-prem solutions (go figure). Being able to switch targets this way would be handy.
评论 #40838956 未加载
systemz11 个月前
Very cool. I needed something like this. Looking at short video in readme - I suspect adding live stats similar to sidekiq would make UI look more dynamic and allow quick diagnostics. Docs for current features are more important though.
评论 #40838199 未加载
deskr11 个月前
Perhaps do a small example application.<p><pre><code> go get github.com&#x2F;poundifdef&#x2F;SmoothMQ&#x2F;models go: github.com&#x2F;poundifdef&#x2F;SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 requires go &gt;= 1.22.2; switching to go1.22.4 go: github.com&#x2F;poundifdef&#x2F;SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 (matching github.com&#x2F;poundifdef&#x2F;SmoothMQ&#x2F;models@upgrade) requires github.com&#x2F;poundifdef&#x2F;SmoothMQ@v0.0.0-20240630162953-46f8b2266d60: parsing go.mod: module declares its path as: q but was required as: github.com&#x2F;poundifdef&#x2F;SmoothMQ</code></pre>
评论 #40838974 未加载
neuroscisoft11 个月前
This is very interesting! The self-hosted aspect is something I&#x27;ll have to consider for certain purposes.<p>My lab also developed an SQS-esque system based on the filesystem, so no dependencies whatsoever and no need for any operational system other than the OS. It doesn&#x27;t support all SQS commands (because we haven&#x27;t needed them), but it also supports commands that SQS doesn&#x27;t have (like release all messages to visible status).<p><a href="https:&#x2F;&#x2F;github.com&#x2F;seung-lab&#x2F;python-task-queue">https:&#x2F;&#x2F;github.com&#x2F;seung-lab&#x2F;python-task-queue</a>
encoderer11 个月前
I think it’s better to say it’s sqs compatible versus an sqs replacement.
rmbyrro11 个月前
I&#x27;d have named it MQLite. Congrats on finishing and delivering a project, this is quite a challenge in itself. I think SQLite can be a great alternative for many kinds of projects.
sgarland11 个月前
This looks great! How were you planning on tackling distributed?
评论 #40838171 未加载
threecheese11 个月前
There is a lot of new ecosystem being built around SQLite in the browser using wasm, as a primary data store or as a local replica for every client, and there are some interesting interactions with crdt and peer to peer applications; does this fit into your business case? Would be interesting to see a massively distributed - via browser embedding - queue, that even uses standard sqs bindings.
michaelcampbell11 个月前
This is really pedantic, totally outside any actual functionality, and may be a me thing, but a trigger for me is seeing left-justified (or worse, centered), columns of numbers (eg: your number of messages column).<p>I&#x27;ll grant it&#x27;s small to infinitesimal, but you asked for feedback.
评论 #40845253 未加载
fbdab10311 个月前
Minor thing - the gif browser demo should have been maximized. The video zooming into the action on what would have fit onto a single screen was distracting.<p>Are there maintenance actions that the admin needs to perform on the database? How are those done?
评论 #40838147 未加载
okr11 个月前
A testcontainer could be useful. I use always this adobe-mock from s3 for my test. Hmm.
评论 #40838119 未加载
pqdbr11 个月前
How would one use this to replace Sidekiq for instance?<p>Rate limiting is something I miss on Sidekiq (only available on the premium plan that I can&#x27;t afford) and the gems that extend it break compatibility often.
lovestaco10 个月前
I was thinking of using Redis Stream instead of SQS, what would be better Redis or SmoothMQ?
leetrout11 个月前
Glad to find out what you&#x27;ve been up to! Very cool and I am excited to see where you take this.
12_throw_away11 个月前
Actually pretty excited to try this, there are so many cases where I need a bare-bones (aka &quot;simple&quot;), local, persistent queue, but all the usual suspects (amqp, apache whatever, cloud nonsense, etc.) are way too heavy.<p>I&#x27;ll probably try poking at it directly through the HTTP API rather than an SDK ... does it need AWS V4 auth signatures or anything?
评论 #40838383 未加载
评论 #40838393 未加载
slotrans11 个月前
Not sure about the goal of providing a hosted service cheaper than SQS. SQS is already one of the cheapest services on Earth. It&#x27;s pretty hard to spend more than a few bucks a month, even if you really try!
评论 #40838510 未加载
评论 #40840898 未加载
评论 #40839973 未加载
TheAnkurTyagi11 个月前
sounds like a great project. How do you handle performance with large message sizes?
xchip11 个月前
Thanks!<p>Loving all these self-hosted KISS stuff :)
skeeter202011 个月前
I love that we&#x27;re seeing a lot of projects apply the KISS principal and leverage (or take inspiration) from SQLite, like this, PocketBase and even DuckDB. An entire generation of developers (including myself) were tricked into thinking you had to build for scale from day one, or worse, took the path of least resistance right to the most expensive place for cloud services: the middle. I&#x27;m hopeful the next generation will have their introduction to building apps with simple, easy to manage &amp; deploy stacks. The more I learn about SQLite, the more I love it.
评论 #40838761 未加载
评论 #40839071 未加载
评论 #40839953 未加载
tjoff11 个月前
SQS: Amazon Simple Queue Service
评论 #40838027 未加载
评论 #40838320 未加载
dracarys1811 个月前
The only reason I use SQS is because how it&#x27;s compatible with other AWS Services, I can have an S3 bucket whenever there&#x27;s an upload I can send an SQS Message and I can have it trigger a lambda.