Hacker News new | ask | show | jobs
by ctlaltdefeat 3393 days ago
There are various mature options for scheduling jobs with dependencies between them.

Why did you choose to write your own, regardless of the language?

1 comments

Good question! None of the existing options that we investigated supported all of our requirements. In particular, all the arrows that can exist in the dependency graph are known ahead of time, but the nodes are not. This means that a job can depend on a job that does not exist yet. (A user can add extra feeds to download, and the merge job should wait for them, even if it had been submitted already.) Furthermore we have a few specific constraints such as “per project, only one of job type x or y may be running at the same time”.