[jawns]

Can you give more details about Visual Regression Testing? If you're comparing screenshots, how do you account for dynamic content?

[mpotter]

Sure! We use a web driver to simulate browsing in an actual client, so Javascript, etc. is executed as it normally would be.

[3am]

WebDriver/Selenium? That's not really visual, like some kind of CV/OCR tool like Sikuli would be. It's a nice tool, I would be more inclined to use it if it supported more robust test environment provisioning (eg, if the pull request is a schema migration) and it gave more control over how to define my own tests, like Sauce Labs.

edit: never mind, I realized you're probably using the built in screenshot features with webdriver and just diffing the images.

[jawns]

But what about content that changes on each page load, like a timestamp or a rotating banner ad? Would that be flagged as a regression?

[mpotter]

Yeah, it's not perfect but right now we're solving this with tolerance thresholds on the image diffs.

[maerek]

Can you share any more details on how the image diffing is done? BBC has a github repo called Wraith that uses ImageMagick and PhantomJS to accomplish a similar task (without the awesomeness of on-demand testing environments). Always curious to learn more about how people are solving the GUI testing problem on the web.

[dbarlett]

There's also Depicted (dpxdt)[1] - more workflow, Python instead of Ruby.

[1] https://github.com/bslatkin/dpxdt