[andrewstuart]

Is your queue bounded?

Does it reject entries when service times are too high?

Your debugging effort may become more predictable when the system measures the time workers take to complete.

I note you say it used to work overloaded. I would argue it probably was having hidden problems. Perhaps ask those people what the acceptable service time is and lock it in by refusing new entries when it is exceeded.

If they want both infinite queue length and consistently acceptable service times then you must add enough work resource to do that.

[bluGill]

Queue is and was bounded, if it gets too large we already logged an error and stopped processing. it is currenty lock free, but the old version had locks (i've tried several versions with and without locks). the bounds didn't change but before it was processing in time even under heavy load, now it isn't.