[thayne]

I don't really disagree with anything you said, but having to run end to end tests for any benchmarking is far from ideal. For one thing, they are often slow,and to get reliable results, you have to run them multiple times, which makes it even slower. And that makes it more difficult, and expensive, to try a lot of things and iterate, and have a short feedback loop. IME writing good end to end tests is also just generally more difficult than writing unit tests of smaller benchmarking code.