[fidgewidge]

Yeah totally. They never had much manual QA. I once worked at Google and part of my job was in a rotation doing server pushes. As part of the push process I'd quickly check out the one paragraph of release notes from the releng guy which would note any new user visible features, and go and play with them on a canary server.

The frequency with which new features point blank did not work at all just blew me away. The problem 100% of the time was that some dev had written a feature given a bug ticket, written some unit tests that covered the code, but never actually brought the serving stack up and played with it themselves. They thought QA meant some test with lots of mocks, not, ya know, actually testing the feature as a user would see it. It was painful to run the servers locally so they just didn't bother a lot of the time.