Not just Postgres.<p>You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED.<p>Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible.<p>You could write a file system based message queue in 100 lines of async python, which I did here:<p><a href="https://github.com/bootrino/arniesmtpbufferserver" rel="nofollow">https://github.com/bootrino/arniesmtpbufferserver</a><p>File system based message queues can be written in any language, extremely simple and, most importantly - zero configuration. One of the most frustrating things about queuing systems is configuration - that includes database backed queuing systems. They can also be fast - I wrote one in Rust which maxed out the hard disk's random write capability well before maxing out the CPU - from memory it beat most of the common queuing systems in terms of messages per second.<p>Not all use cases for queues need to be able to globally distributed messages queues with the sort of guarantees needed for financial transaction processing. I would suggest to you that in fact most queues out there are used as outbound SMTP queues, which are then over engineered to use something like Celery, which is a nightmare to configure and debug.
A controversial, but a very pragmatic take.<p>Queues are great for semi-infinite scalability, but you rarely need it.<p>There's numerous subtle benefits to using db compared to regular message queues that are often overlooked.<p>Being able to delete, reorder, or edit specific messages can be a lifesaver when things go wrong.<p>Most common issue with queue-based systems is getting overwhelmed with messages. Either your consumers went down. Or your producers had a bug / executed too frequently.<p>Being able to recover with a simple SQL query is a blessing.
"The write path. We built a daemon that would select log entries that were older than two weeks, copy them into a file in S3, and delete the rows from the database"<p>Seriously... How can you ever consider saying "a rdbms is just fine as a kafka alternative" under those conditions ?
> Postgres: a better message queue than Kafka?<p>"We didn't actually evaluate Kafka for this use case, because we know Postgres, so we used that."<p>Weird article angle.
File this under "If a headline is asking a question, then the answer is NO." Honestly, I'm not even sure what point the author is trying to make besides "anecdotally and at low scale, unusual-for-the-purpose technology X solved problem Y." A near-infinite number of bad patterns can solve problems along happy paths and resolve plenty of edge cases to boot. 99.9% availability was a goal post here? There are systems where 7 9s is unacceptable. The author didn't even provide data backing the measurement goals.
Something I think needs to be stated: Kafka is not a message queue. If you need things like work stealing, you're going to have a bad time. It doesn't seem to apply in this case, but the terminology misuse is going to lead some poor developer astray.<p>It seems like they only tried postgres? so they didn't compare it with kafka at all....so yeah, you can make a message queue or log table in a rdbms.
You can spew messages into sqs then have a lambda on sqs sending to pg.<p>Funny, we were going to use kafka too, but just sending to mysql worked just fine <shrug>. One day that will change, obviously.
I don’t think this article made a compelling reason not to use Kafka. In fact it may have made the opposite point.<p>Wouldn’t it have been easier to just use Kafka?
I don’t want to doubt Postgres’ effectiveness as a message queue (however difficult it may be) but I have to wonder if maybe they were doing something weird or wrong in the consumer for it to be ineffective?<p>Maybe it’s not that Postgres is a better than Kafka as much as Postgres is a better solution for what they were trying to do and use Kafka?
“ Today these are not major problems for us, and frankly, I’m not sure how we’ll tackle these problems. Maybe we’ll grow multi-region support and buy really big Postgres boxes. Maybe we’ll move to sharded Postgres. Maybe we’ll switch to Kafka or some other distributed queue. ”<p>LOL
Since they're already doing offload to S3 for older log messages, it seems like they could do this from the get-go and save a bunch of storage and interzone transfer costs in RDS. Depends on the log message sizes though of course.
There is also commercial products like NServiceBus that can use SQL Server as a connector to provide storage. I always used to think about doing this in the database as poor mans message solution. Perfect for small projects.