[bri3d]

Automated tests are very poor at capturing the nuance of user interaction, and I find that they frequently are not exhaustive or watching the video shows only a "happy path" and doesn't expose functional deficiencies in a feature. They show that a feature works, but not that it works correctly or especially not that it works _well_.

For straightforward regressions and minor tweaks I am usually satisfied to see a video and test automation, but for any kind of new functionality I strongly advocate pulling the code and actually playing with it.

Depending on your company structure, product managers can help with this functional review as well, although this requires working tooling and org structure in a way that doesn't exist at some companies.

[quixoticaxolotl]

It sounds like what you want is a beta testing phase once a feature is released, not a per-PR play phase.

I've had companies host release parties before, where everybody downloads the product and plays with it to identify glaring issues. It's a decent approach, but you'd get way better results having a dedicated testing team whose job is to test the software from a UX/UI/contract functionality if that's what you care about.

[bri3d]

I've used the release party approach, and it works, at a high cost. I've also worked with really good QA teams. The best ones don't really test contracts, they test the contract itself - that is, instead of testing that the UX behavior matches the "expectation", they pressure test that the "expectation" matches what a real user might do.

But even in these places, I still find value in having a second set of eyes pull down a PR and see if it makes sense. With good tooling it should only take a minute or two (and without good tooling, investing in this tooling usually helps a ton of other places too). This saves time in the ultra-expensive "30 sets of eyes pull down the product and use it" party phase, and frees up QA to be even more effective at pursuing edge cases.

Plus, again depending on the company, often fixes forward from mainline are kind of expensive, especially if you work in B2B / Enterprise where change management often works on a cadence. Fixing the feature on a branch can be a lot cheaper and easier than fixing it once it's in the wild.

Anyway, it all depends on the team and their process, and I'm far from a zealot about much of anything, but I do really encourage pulling down PRs and clicking through them as a matter of course, regardless of what additional mechanisms are in place.

[antifa]

I'd expect tests to have assertions that all the expected 401/403/404/422 responses actually happen and have the correct response messages.

[ryan_lane]

You're describing what QA's function normally is.