Hacker News new | ask | show | jobs
by akamaozu 1170 days ago
re: Prefetch and Reconnection Issues

- clear case of misconfigured instance (prefetch) and a bug somewhere in the stack (reconnection issue)

- prefetch behavior sounds like they receive a message, ack it then process it

- i wouldn't recommend ack before processing, because you become responsible for tracking and verifying if the worker ran to completion or not.

- work then ack is the way. the other way around ignores key job processing benefits like rabbitmq automagically requeueing messages when a worker crashes and failure related logic like deadletter queues.

- the trick i've started leaning on with rabbitmq is giving each worker their own instance queue (i call it their mailbox).

- when a worker starts a job, it writes the job id, start time and the worker's mailbox to a db. any system can now look up the "running job" in the database, know how long it has been running and can even talk to the worker using its mailbox to inquire if it is running and if that job state in the db is accurate.

- happy the writer and team found what works for them. ultimately, what you understand best would serve you better, so they made a good choice to lean on their strengths (postgre).

1 comments

Other comments point out that your proposed solution is probably their problem and they should do a quick ack instead.
I don't agree with their assessment, but also not really interested in putting effort into debunking it.

Just occurred to me there's a different possibility:

The extra connection bug mentioned could be from a rogue thread/process/code-path connecting to the queue and fetching an item, but somehow not doing the work.

Verifying this likely requires access to the codebase, so treat this as little more than idle speculation.