[ryze20245]

I feel like a more effective way to use cron is just to dispatch jobs into a queue that will perform the actual processing. And not to do the processing within the cron scripts themselves. That way the load on the cron is light and the heavy lifting is done by your queue/worker system.

[getoffmyyawn]

This works great in my experience.

A lifetime ago I scaled up cron jobs for a client with Gearman. Using cron to trigger jobs on the Gearman server and the pool of runners to do all the work. This proved to be so reliable they still use the system today, over 10 years later.

[djboz]

Really cool! For the Gearman workers, did you load jobs dynamically? Or, would you have to re-deploy for new jobs/updated jobs?

[getoffmyyawn]

As I recall, all the jobs are checked into a repo that is deployed to all the runners, which each start gearman workers for their assigned role.