[Eridrus]

The thing keeping us from CD is a lack of faith in our automated testing, mostly because we don't have enough automated testing, but partly because our problem domain makes it very hard to automate the testing that we are really concerned about doing. Part of this is definitely lack of testing culture though.

[brendanp]

Yeah - I'm sympathetic. It's often difficult to sell people on the notion that integration testing might be just as expensive to implement as the service/feature that you're building in the first place.

It sounds like this is partly an education issue - are there specific areas within your problem domain that more tooling would actually help with?

[Eridrus]

Our fundamental problem is that we have a Typescript (<3) analytics payload that has to deal with lots of weird JS execution environments that we fundamentally can't recreate in a lab, which means we have no idea if collection code is working as intended until we run it in prod and have a manual look at the data.

But I started thinking about this in a bit more detail, and the things that make our problem domain hard don't necessarily have to block doing fast releases, since the failure scenarios that we want to block should largely be automatically detectable.

The parts that we haven't been able to automate though are around verifying that our payload, does not have any user visible impact on sites we run in, including DOM elements we want hidden definitely still being hidden, verification that we don't trip any SSL or other warnings, and we don't generate console warnings from the browser, etc. So if Selenium was not a complete hack job and had more knowledge of the browser chrome/UI, we could probably fully automate it.