[herge]

Maybe the spry takeaway is that you should write tests for things you want to make sure they will not break.

Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test.

However, your business logic, or network protocol routine, those should not break even if you heavily refactor or add new features (especially business logic where a broken behaviour might seem correct), so those need to be heavily tested.

If it is hard to test the juicy parts like business logic without also dragging in the UI, different OS/platform/db parts, etc, then you should look at how your application is structured and if it is really optimized for writing good tests.

[gav]

> you should write tests for things you want to make sure they will not break

You should also write tests for things that are already broken before you work on the fix so that you can be sure it's actually fixed. Basically the red/green/refactor cycle[1] from TDD.

[1] http://www.brendanconnolly.net/test-driven-testing-red-green...

[jasode]

>Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test.

For brevity in the previous comment, I didn't fully flesh out the background on why fragile UI tests get created. It happens accidentally.

What sometimes happens is the the UI tester uses a "macro recorder" to record mouse movements and clicks. But then, a programmer shifts the position of a zipcode field by one pixel which throws the script off because it expected UI elements in a different spot. Fixing the broken UI tests is time consuming and can leave a bad impression that tests create a lot of effort for very little payback.

The return-on-investment of UI tests depends on the business circumstances. I'm guessing Boeing and Airbus have automated UI tests that sometimes break when programmers change things which causes rework. However, the pain of fixing the UI tests and keeping it sync'd with the UI code is worth it for avionics software.

[strken]

Visual diffing in UI tests is something that actually can be solved with more code, see https://github.com/americanexpress/jest-image-snapshot, https://github.com/gemini-testing/gemini, or https://github.com/BBC-News/wraith.