|
|
|
|
|
|
by tmcneal
1597 days ago
|
|
|
I'm not sure if any of these are pertinent to your tests, but these are the issues I see most often that cause flaky tests: - Hard-coded waits in your code, like "Thread.sleep(1000)". A better alternative is to replace hard-coded waits with something that waits for an element or value to appear on the page. i.e. click on a button and wait for a 'Success' message to appear. Puppeteer and Playwright both have good constructs for doing this. - Needless complexity in the tests. Conditionals in particular are a code-smell and indicate there's something needlessly complex about the test. - No test data management strategy. The more assumptions you can make about the state of your application, the simpler your tests become. Ideally tests are running in an environment that nothing else is touching, and you're seeding data into that environment before tests run. I personally don't believe in mocking data in regression tests since that quickly becomes hard to manage. We spend a lot of time thinking about these issues at my company and wrote a guide that covers other common regression testing issues in more detail here: https://reflect.run/regression-testing-guide/ |
|
|
We don't do any timed waits, all of our waits are for an element or value to appear, but these waits never complete sometimes, non-deterministically. We then added a long 5 min timeout on these waits because we know the test will never complete at that point. It's always fine in manual testing though, and if we don't run the browser in headless mode and watch it work. Very frustrating.
Sometimes the HTTP requests themselves timeout after a few minutes, but this never happens in manual testing either. That's actually the most common issue these days, and this happens non-deterministically too. This is what I meant by "flaky".