Hacker News new | ask | show | jobs
by hipadev23 947 days ago
> Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering.

Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.
3 comments
lazyant 947 days ago
> I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.

I'm also very familiar with jobs and I have used the usual tools like Redis and RMQ, but I wouldn't make a blanket statement like that. There are people using RDBS as queues in prod so we have some counter-examples. I wouldn't mind at all to get rid of another system (not just one server but the cluster of RMQ/Redis you need for HA). If there's a big risk in using pg as backend for a task queue, I'm all ears.

link
toolz 947 days ago
as far as I'm aware the most popular job queue library in elixir depends on postgres and has performance characteristics that cover the vast majority of background processing needs I've come across.

I wonder maybe if you've limited yourself by assuming relational DBs only have features for relational data. That isn't the case now and really hasn't been the case for quite some time now.

link
qaq 947 days ago
Postgres based job queues work fine if you have say 10K transaction per second and jobs on average do not take significant time to complete (things will run fine on fairly modest instance). They also give guarantees that traditional job queues do not.
link
Rapzid 947 days ago
Probably order of magnitude more or perhaps a multiple of that depending on the hardware and design.

In theory an append-only and/or HOT strategy leaning on Postgres just ripping through moderate sized in-mem lists could be incredibly fast. Design would be more complicated and perhaps use case dependent but I bet could be done.

link
qaq 947 days ago
Yep that's why I specifically mentioned "fairly modest instance" on reasonably fast box you can get magnitude more. You can partition the tasks table to reduce the number of rows skip locked has to run through to grab next task.
link