[JamesBaxter]

I’ve tried building in load tests to releases and it’s been a headache so far: Test environments cost as much as prod environments suddenly (using premium versions of infrastructure to meet requirements)

Pipelines take a lot longer to run (because the load test environments are expensive we have one or two and serialize test runs on them to stop test runs stepping on each other, but the teardowns are slow, especially on failed runs)

To keep the test run fast we limit them to about 5 minutes but this creates unrealistic expectations of how fast our code would actually need to scale.

But it is nice knowing which changes have broken the load test. Hunting them down is so painful.

What I’d to have is proper metrics showing how close we are to our Non Functional Requirements over time to make it easier to sell “Performance stories” ahead of breaking the NFR rather than just fixing it as we break.

[ericb]

Hi James, I'm working on a load testing tool product. Any chance I could talk to you about your headaches, and how you do things? I'm trying to learn what people are doing now, and where the pain is. Maybe you could tell me if my ideas are crazy--I am taking a different approach than what's out there. :-) I'm at ebeland on gmail.

[wdb]

I like to run over night. It's not superfast feedback but still a reasonable timeframe