[Nican]

I have been toying with the idea lately of using a transactional database (like SQL) to manage some of the very important queues.

Using a transaction to retrieve an item from the queue, and locking the row using "SELECT FOR UPDATE" and "SKIP LOCKED". Such that the row gets locked on read, and several workers can read from the table at the same time. Within the same transaction, other work is done, and everything gets committed to the database as a single atomic operation.

CockroachDB (a consensus/raft distributed database) recently added supported for SKIP LOCKED, but I still have yet to work on this idea.

[rowls66]

I have worked on a system that took exaclty this approach for ~17 years. The database was Oracle, at the time we started 'SKIP LOCKED' was not even a documented feature of the Oracle DBMS. It is now. The approach worked quite well for us and happily working today at several large banks. Also, Oracle sells what I think they call AMQ (Advanced Message Queing) that provides a messaging API but uses the DBMS for storage. No idea how it performs relative to dedicated persistent messaging solutions, but I would guess that it probably good enough for many workloads.

[kevincox]

It can definitely work. It is a pretty good way to avoid the distributed systems problem by avoiding having a distributed system.

Of course there are other concerns with using a database as a queue (mostly at high throughput) but for most cases it will work well.

[scarface74]

Using a database as a queue. What could possibly go wrong?

I guess everyone has to make this mistake once in their career.

Funny enough, when I searched for “database as a queue”, my own comment from four years ago came up as the fourth result.

https://news.ycombinator.com/item?id=18774559