[w0mbat]

Various mass-market desktop apps that I've worked on have had a test system like this, where tests will screenshot the new build in various states and the pictures are compared with known good reference screenshots from a previous build. If a comparison fails, QA will check that the change is intentional and either update the reference image or file a bug.

Even quite minor OS updates can cause the tests to fail en masse, because of a global OS change to the system font or button design or whatever, which is a shame as that is just the point where you want to see what the OS change actually broke (like one of your windows now comes up offscreen, or your help screen is now always in Icelandic).

With app testing you can't restrict the detection to the window content area, as a bug could for example give the main window the wrong kind of title bar, or make it draw its default title in the wrong localized language, and you would want to detect that.

Then when making UI changes you need a mechanism for marking which comparisons are intended to fail after this change, and will need to be automatically regenerated for the next build. I don't think I've ever worked on a project that got this entirely automated, and it resulted in a lot of work for QA. On a complicated app like a web browser it is a really valuable system though.

[pytester]

>Even quite minor OS updates can cause the tests to fail en masse

I found it didn't even take that. Sometimes indeterminism in the code you wrote or indeterminism in code you can't even control will cause tests to fail en masse.