Hacker News new | ask | show | jobs
by parliament32 2220 days ago
Unfortunately, it's still missing the main cron-killer feature: randomized intervals.

I have 200 servers that I need to run some task every 3 hours. But I don't want them to all run at the same time. I need options to either "start this task at some point randomly in the next 3 hours, then every 3 hours thereafter" or "run this task randomly in every 3 hour block", without resorting to manually tweaking hour/minute run times on every server.

4 comments

As a workaround, sounds like you need a deterministic pseudo-random number generator.

Find a value that is unique to each server (e.g. the UUID of the / partition?) to seed your PRNG, and then generate the first pseudo-random number. If you do it right, it will always be the same number.

Modulo this number against the number of (seconds|minutes|hours) of random delay you want applied, and apply a sleep timer to the start of the job based on that result.

It's dirty, but it ensures

* each server gets its own random interval from the start time

* that random interval never changes

* the server runs the job on a regular schedule

There are probably better ways of doing this, but this is the first thing I could think of that doesn't require any crazy libraries and can be done using standard linux utils that are on all machines.

I'd love to hear any better ways that people have!

> Find a value that is unique to each server (e.g. the UUID of the / partition?) ...

FWIW, that's pretty much what /etc/machine-id is for.

I'd imagine that systemd timers would work here, and I would imagine would be less work than having to have 200 jobs in some 3rd party service.

A few ideas off the top of my head: write a unit template and use the instance parameter to designate an offset; or write a single unit template with the `OnCalendar` property set to `*:00:00/3:00:00` and `RandomizedDelaySec` set to (10800) (3 hours in seconds).

Of course, the same basic effect is also possible with traditional cron. Certbot's Debian package has a pretty basic example of how to do basically what you want: have the job set to run every 3 hours, and then have the command be something akin to: `perl -e 'sleep int(rand(10800))' && <your command>` - it's the exact same high level logic as the systemd approach, but you're inserting the delay yourself using a `sleep` call.

I'd personally very much go for the systemd approach but either should work.

In my experiences, the "standard" way to do this is to simply add a random sleep before running the task.

For example, to run a task at some point between midnight and 3 a.m., you can use this cronjob entry:

  0 0 * * *  sleep $(( $$ \% 10800 )) && /foo/task.sh
Alternatively, systemd has the "RandomizedDelaySec" option.

(Note: 10800 == 60 x 60 x 3, the number of seconds in three hours.)

You may want to checkout fcron

http://fcron.free.fr

It has the every N since last run option.

And jitter and random features too