[omrvp]

Thanks all for the comments about the PinLater post - great to see this discussion! Wanted to clarify a few things:

* Yes, we did evaluate a few open source solutions before deciding to build in-house. We couldn't at the time find any solution that met all the requirements that we had: scale, scheduling jobs ahead of time, transactional ACKs, priority queues, configurable retry policies, great manageability (e.g. ability to rate limit a queue online, inspect running jobs, review failed job stack traces). [update: also support for clients/jobs in multiple languages: Python, Java, Go, C++]

* Note that we were building a v2 system. Not improving substantially on the existing system on multiple dimensions would not quite make it worth the cost of doing large scale migration of tasks. We had to meet all the requirements, could not compromise on missing a few.

* Sorry I didn't mention explicitly in the post, but we do plan to open source PinLater, hopefully later this year.

Happy to answer any other questions or specifics!