[devman]

Strange article.

Does one really need to do data analysis to see that it is simply natural for large tests to be more flaky? All else being equal large tests perform more operations. If on average the probability of any single operation to fail is constant (fair assumption given the large number of tests) than the test that performs more operations has higher probability to fail (and no it is not linear as shown in the article, it is asymptotic approaching probability of one). The "analysis" of tool types are strange too. Webdriver tests are more flaky because of inherent nature of UI tests operating on less stable interface (compared to e.g. typical unit test which usually deals with a pretty stable contract).

I used to spend quite some time on test stability issues in my previous team and wrote a couple posts on the topic and how to successfully go from unstable to stable test suite:

https://blogs.vmware.com/management/2015/09/automated-tests-...

https://blogs.vmware.com/management/2015/09/automated-tests-...

[Edit: formatting, spelling]

[hugs]

It's not strange​ at all. I appreciate the article's analysis. As the founder of the Selenium project, I frequently hear people blame Selenium for their problems. Sure, Selenium isn't perfect, but it seems like it is so much easier for people to blame their tools instead of questioning other things. But I understand why people do it. For a typical software developer, without an obvious cause for flakiness, the apparent randomness of a flaky test tends to make it easier (and plausibly justifiable) to "shoot the messenger".

[vosper]

OT, but thank you for Selenium!

[hugs]

Thanks, although credit these days rightfully goes to the huge multi-company and org team effort. (In other words, at this point, there are tons of people to blame for those flaky tests!)

[stygiansonic]

The reasoning behind why large and Webdriver tests ought to be more flakey is sound. However, I don't see why having empirical data and doing the analysis is "strange".

Doing studies and collecting data should be how any assumption, seemingly reasonable or not, is proven right or wrong.