[stickfigure]

40,000 per second is waaaaay beyond where you should use a dedicated queuing solution. Even dedicated queues require tuning to handle that kind of throughput.

(or you can just use SQS or google cloud tasks, which work out of the box)

[pritambaral]

I hit 60k per second in 2020 on a 2-core, 100GB SSD installation of PG on GCP. And "tuning" PG is way easier than any dedicated queueing system I've seen. Does there exist a dedicated queueing system with an equivalent to EXPLAIN (ANALYZE)?

[stickfigure]

If that's true, you managed to do much better than these folks:

https://softwaremill.com/mqperf/

Maybe you should write a letter?

[winrid]

It's possible the person you're replying to wasn't using replication, so it's entirely different. Those folks also used "synchronous_commit is set to remote_write" which will have a performance impact

[pritambaral]

This is correct. My use-case was safe with eventual consistency, so I could've even used `synchronous_commit=off`, but I kept it to 'local' to get a baseline. Was happy with the 60k number I got, so there was no need for 'off'.

But I think the biggest reason I hit that number so easily was the ridiculous ease of batching. Starting with a query to select one task at a time, "converting" it select multiple tasks instead is ... a change of a single integer literal. FOR UPDATE SKIP LOCKED works the same regardless of whether your LIMIT is 1 or 1000.

[kstrauser]

I worked at a shop that had to process about 6M RPS for 5 seconds at a time, once a minute or so. That looked a lot like a boatload of Python background threads queueing work in memory then flushing them out into Cassandra. That was a fun little project.