[pkkim]

Yeah, you make some good points, orchestration has been historically painful -- we've personally seen the headaches that come with scheduling on slurm and lsf; I'd guess some of the most thorough bikeshedding in history has been around tinkering with slurm's multifactor scheduling logic. We're trying to not to re-invent the wheel with orchestration, and we're in the midst of building interfaces to hook into slurm, instead of replacing it entirely :)

As for buck2, we decided to go with it for a few reasons:

More forgiving with gradual adoption, from our experience -- running non-sandboxed actions in bazel is a pain, buck2 has been much easier to plug into existing flows.

Buck2 installation is easier, and by extension, is simpler to embed into our test runner.

Respectfully, bazel's implementation is a monolith beyond comprehension -- if we want to modify buck2, and package our own fork, we have confidence that we could do that.