I dug into the internals of the local, SQLite version of this just now and wrote up some notes here: <a href="https://til.simonwillison.net/deno/deno-kv#user-content-deno-queues-27th-september-2023" rel="nofollow noreferrer">https://til.simonwillison.net/deno/deno-kv#user-content-deno...</a><p>The most interesting detail is probably the schema they're using for that:<p><pre><code> CREATE TABLE queue (
ts integer not null,
id text not null,
data blob not null,
backoff_schedule text not null,
keys_if_undelivered blob not null,
primary key (ts, id)
);
CREATE TABLE queue_running(
deadline integer not null,
id text not null,
data blob not null,
backoff_schedule text not null,
keys_if_undelivered blob not null,
primary key (deadline, id)
);
CREATE INDEX kv_expiration_ms_idx on kv (expiration_ms);</code></pre>
I think what Deno is aiming for here is actually very forward thinking.<p>I'm using Go for the first time this year and some aspects of the language are very C like. Of the list of things that isn't C like (e.g. GC allowing one to return what look like stack allocated pointers from functions) there is the obvious inclusion of things like `map[string]string`. I bring this up because it struck me that inventing a language in 2023+, it would seem almost insane not to have built in syntax for the language to handle map types.<p>And so it seems logical that a web-server focused eco-system would start to garner libraries and even language syntax (maybe one day) for the kind of primitives we frequently use on web-servers. I mean, I can't even recall the last time I worked on a distributed web server that didn't have a KV store as a cache, or a locking mechanism, or even an ad-hoc queue. A distributed system without a KV store feels vaguely isomorphic to a computer language without a map type. It is such a Swiss army knife kind of technology.<p>One potential issue is that Deno is going alone down this path. Currently, I don't feel confident that the new features they are adding will be available on competing platforms. Even if they open source the API (it is on a `Deno` namespace), I'm not sure it will just work on AWS if I wanted to switch out FoundationDB for e.g. Redis.<p>For that reason, I feel I want to avoid Deno. Even if the syntax and exposed features are really attractive, I'm worried about becoming locked in and then having to do a lot of surgery to code to make it deployable on multiple cloud infrastructures. That is a requirement from a lot of clients. E.g. maybe I want to sell to Oracle or Salesforce one day but they mandate I have to run my systems on-prem. I'm now on the hook trying to figure out how to adapt whatever available KV store they have to the features I'm using from the `Deno` package.<p>It is a double edged sword. Maybe they will succeed in pushing forward this vision to a broader audience. For now I'll probably remain cautious.
> Leveraging public cloud infrastructure has traditionally demanded sifting through layers of boilerplate code and intricate configurations, often monopolizing a significant chunk of the developer’s time and energy.<p>I don't buy this line of reasoning. At the end of the day we are building infrastructure that needs to be reliable. Spending 30 minutes to set up an SQS queue (proven technology) doesn't sound as bad as putting all my trust in a toy queue that Deno built "on top of SQLite / FoundationDB". Is the 30 minute setup cost and added "developer experience" really worth the risk? How often are you setting up queues anyway.
> at least once semantics.<p>> user code.<p>A past life has taught me that users will never properly understand at-least-once semantics. Anytime you redeliver messages, you will get a flurry of user complaints and breakage.<p>Either you do the impossible and invent a way to do exactly-once semantics. Or you should always redeliver 0.1% of all messages, just so that users don't come to depend on messages being delivered once.
I'm currently using Deno deploy and found it to be fantastically performant and dumb simple for my lone wolf project. I'm experienced in AWS development in larger teams and it is nice to see a move away from complexity for a change where you can easily set something up without having to think about setting it up at all. The DNS stuff was just dead simple as well and automatic ssl certs was super nice. I have 0 complaints for what this is trying to be and am excited for the road map.
Unless I'm missing something, it looks like each Deno.openKv() instance only gets a single queue.<p>For the local version you could get multiple queues by calling Deno.openKv("db-2.db") with different SQLite file paths each time, but that feels like a lot of overhead for a pretty common need.<p>I guess this is a Deno architectural style thing - maybe when you build complex apps on Deno it's expected that you'll have a microservice style architecture where lots of different scripts work together, each of them with their own KV store and hence their own queue?
Pricing?<p>I thought Deno was some sort of node.js replacement. What am I missing and can I use this either locally and/or self hosted without paying for it?
(on the off chance the devs read this) I can see pain in the future for the currently excellent KV ergonomics around access control --- is the solution just "implement it in userland and don't write bugs" or is there anything planned?
I'm excited about the recent Jupyter support and this queues.. Super cool stuff.<p>But I won't write servers in Deno and put them to production in my own servers. (I don't like serverless)<p>You see, Deno(and Bun)'s business model is built around serverless (Deno deploy). And Deno deploy is ANOTHER runtime.<p>It's clear with KV and queues... Locally and if you self host, you have a barely working version backed by another tech...<p>This means you're self hosting a version of Deno that's different from the majority of the Deno customer base.<p>How many people will use Deno to run servers on their own hardware? Since that number is low, then I'd rather use Node.
I wonder how this differs from BroadcastChannel[1] which Deno already implements and DenoDeploy seems to support.<p>[1]: <a href="https://docs.deno.com/deploy/api/runtime-broadcast-channel" rel="nofollow noreferrer">https://docs.deno.com/deploy/api/runtime-broadcast-channel</a>
I don't know if Ryan and the rest of the Deno team are lurking here, but given "Anchored on the robust capabilities of FoundationDB" what I think would be absolutely fricking cool is to get <i>full access</i> to the underlying FoundationDB engine, which would surface the "model any data model you want the way you want it" capabilities of FoundationDB directly to Deno developers.<p>In case anyone wonders what this would make possible out of the box: <a href="https://apple.github.io/foundationdb/design-recipes.html" rel="nofollow noreferrer">https://apple.github.io/foundationdb/design-recipes.html</a>
I actually do quite like the overall idea of having a unified API at the language level that a (configurable) runtime can provide implementations for. Sure, in some cases an API might not be easy to abstract over due to underlying concerns, but in a lot of cases it does work.<p>It does show a level of understanding by the language creator that they know how the language can and probably should be used - I like how KV defaults to SQLite locally and takes on new meaning in a hosted environment. I haven't seen, but as long as you can actually override the behavior to use your own KV/queue technology (so long as it satisfies the interface) I see absolutely no issue with this.<p>There is a huge benefit where the code always looks the same for common things we all need to do. I wouldn't mind seeing other languages take an overall stab at this so there is some choice other than JavaScript.
Very timely... I'm almost finished porting WakaQ (my custom background task queue to replace Celery) from Python into TypeScript to power background tasks for a new Next.js website. I'm using T3 not Deno so I wouldn't have used Deno Queues, but it's a core part of every web app so makes sense they would build this into their stack.<p>Some questions:<p>* Looks very powerful, but at the core is it just one single queue? Does Deno.openKv() create a new queue or re-use the same queue every time it's called?<p>* Usually I need multiple queues of different priority, so when bottlenecks happen the highest priority jobs run first.<p>* Maybe they guarantee infinite worker capacity so you don't need queue priorities? No bottlenecks if you can pay for the jobs you enqueue?
Under the hood, does DenoKV implement an abstraction that specializes locally to SQLite and to Deno Deploy in the cloud? Or are DenoKV local and DenoKV cloud two separate products with a common public API?
What happens to the mantra that "database as a queue is harmful"[1]? Of course Amazon teams internally use DynamoDB for all kinds of queues, which implies that we get to do a lot of things easily if we get a super robust storage solution. So, I guess the question is really whether FoundationDB can be used as a backend for queues.<p>[1] <a href="https://www.google.com/search?q=Use+database+as+a+queue+is+harmful" rel="nofollow noreferrer">https://www.google.com/search?q=Use+database+as+a+queue+is+h...</a>
I like how updating the data and enqueueing a message can be part of a single transaction. That's indeed pretty powerful.<p>A bit of an aside, but not sure we want at-most-once delivery for email.
Isn't deno running locally no your self-hosted instance? How are they charging?<p><pre><code> Enqueuing a Message: Each enqueue action translates into a KV write operation.
Receiving a Message: Every received message entails a KV write, and a single request charge.
</code></pre>
I would be really curious to know. Are they only talking about Deno Deploy in the cloud, with FoundationDB? Is this "open core"?
> Leveraging public cloud infrastructure has traditionally demanded sifting through layers of boilerplate code and intricate configurations, often monopolizing a significant chunk of the developer’s time and energy. Our goal is to distill these intricacies into user-friendly primitives, enabling developers to design, refine, and launch their projects with unmatched speed.<p>I really like this
This is amazing syntax. If you are looking for a self-hostable version of this, that can run deno but also python, go, and bash, with primitives similar to airflow and more (retries, cache, suspend, approval steps), check out windmill: <a href="https://github.com/windmill-labs/windmill">https://github.com/windmill-labs/windmill</a>
<p><pre><code> With grand promises rode the ploy
Venture capital, ahoy!
We make a lovable product to capture audience
Vendor lock them at the earliest convenience
Hey look at our awesome developer experience
Add ANSI and emojis and comments saying it’s genius
Some next level tech, plebs just don’t get it
Next round of funding, we’ve already spent it
I took the big dough, got puppets and framed it
R-O-I is bad though but layoffs gon’ fix it
Never mind the drama *cough* are ya gon’ install it
Run it, ship it, kill yourself with working on it
Entangle your business with it
Tell the client to deal with it
Bring it to the big wig
Get ’em to sign off on it
Crown your own sh*t
Dogfood is tasty, ain’t it</code></pre>
I feel its odd for a compiler/language toolchain to have a cloud offering. I understand the motivations but 20% of the article was about their Deno Deploy product and how to calculate API costs...<p>I understand its only when using their cloud and there is a local implementation, but I just couldn't imagine using a feature in LLVM, for example, where if you were to upload the binary to a specific cloud it would use a different implmentation that costs behind the scenes.<p>It just feels odd to me. I guess it's kind of cool. Maybe different clouds can implement the underlying KV and then this becomes cross-cloud with different underlying benefits without changing the code. But I'm not entirely sure how I feel about this.
On the topic of Deno's cloud deployment and Deno KV, does it offer a reasonable user story for caching? Or is that something you're expected to handle yourself (for now)?
I never liked Agenda (Persisted delayed task queue using MongoDB). This example[1] using Deno Queues & KV is easy enough to be a replacement for Agenda.
I don't want my programming language to "implement" task queues and charge me for it. What is this<p>edit: people seem to be getting hung up on the "runtime" v "programming language" distinction. I'm not sure why--it's weird to me that this is "part of the language + runtime" at all. Clearly, reasonable people can disagree about that, but hopefully on points more substantive than pedantic<p>If tomorrow, someone started a for-profit company making a faster Python runtime and introduced features like this, it would be weird, and it would feel as pointless to me as Deno is