[gnagatomo]

>the most common result (by a large margin) of a test failing was someone would update the expected file/image so it would pass with the new visuals Previous solutions that my team developed ended up like this. We found the problem can be split in two:

1. Figuring out if some change is actually wanted/expected

We found that doing a d-hash + hamming distance (levenshtein) is a very good way to handle tolerance, as it will ignore most text and subtle layout changes, but unlike most examples the screenshot is scaled down to 200px wide (this number is arbitrary, we're still figuring out how much tolerance it does provides).

2. How to handle this changes

This is the hard part. For now we are only selecting the images that failed in the previous step and by using some jenkins plugins they are presented during the pipeline side by side (old, diff, new) and are reviewed manually, but the versioning process is automatized and the results are stored in the changelog.