[hinkley]

This is an oversight that lands in circle of “the things I know that are difficult to sell because everyone else is just ignoring it.”

Capacity planning applies to tests. As your test count goes up your budget per test goes down. Every CI tool should plot build and test duration over time and only a couple do.

There’s a reason most test frameworks can mark slow tests. You need to not only use that but ratchet down over time. Especially when you get new hardware.

I haven’t run benchmarks lately but my old rule of thumb developed over many projects and with several people better at testing than I, was a factor of eight for each layer in the testing pyramid.

That certainly puts a lot of runtime pressure on the top of the pyramid, but that’s by design. You don’t want people racing to the top, because that’s how you get a cone.