[nikita2206]

The first one is basically the messaging system and a smart broker. I’m not aware of any brokers that would take into account how busy a particular node (they usually just use round robin approach) but surely the concept you outlined is far from being a definition of over-engineering, cron is almost always worse because it is hardly scalable (it is with some crutches and until some point when it becomes unscalabel again, and it introduces a consistent delay).

[elvinyung]

It's less of a message broker and more of a scheduler, although considering that both of them have some kind of queue at its core, the distinction is probably more superficial than it seems (a FIFO job queue is basically a kind of scheduler).

Spark has a scheduler that takes data locality into account (although I don't have a ton of experience with it and have no idea how well it works on a fine-grain scale). That itself is probably inspired from Borg and MapReduce, both of which have some kind of data locality-based job scheduling logic.