[brendanp]

Yeah - I'm sympathetic. It's often difficult to sell people on the notion that integration testing might be just as expensive to implement as the service/feature that you're building in the first place.

It sounds like this is partly an education issue - are there specific areas within your problem domain that more tooling would actually help with?

[Eridrus]

Our fundamental problem is that we have a Typescript (<3) analytics payload that has to deal with lots of weird JS execution environments that we fundamentally can't recreate in a lab, which means we have no idea if collection code is working as intended until we run it in prod and have a manual look at the data.

But I started thinking about this in a bit more detail, and the things that make our problem domain hard don't necessarily have to block doing fast releases, since the failure scenarios that we want to block should largely be automatically detectable.

The parts that we haven't been able to automate though are around verifying that our payload, does not have any user visible impact on sites we run in, including DOM elements we want hidden definitely still being hidden, verification that we don't trip any SSL or other warnings, and we don't generate console warnings from the browser, etc. So if Selenium was not a complete hack job and had more knowledge of the browser chrome/UI, we could probably fully automate it.