Does anyone know if this can be used for effective performance testing? It’s not clear from the Playwright documentation, but I know Selenium (a similar tool) does not recommend performance testing.
We use both Testim.io and Playwright for performance testing.
I work at Microsoft but at a completely unrelated team and while I know some of the folks who work on playwright I have no official affiliation and my opinions represent my own and not my employer's.
Playwright is great for this sort of thing! All we do is run the same test with two variants (before/after the change) and run a student t-test (well, a welch t-test but close enough).
It's ±50 LoC and as long as you're fine with running the test enough times to get statistical significance Playwright works quite well for this
It depends exactly what you mean by performance testing. The problem with a lot of UI tests is that you might need to add artificial slowness to make the tests more likely to pass. Even waiting for something to appear can take longer than it would in a browser. If you want a basic number to see regression, sure you can use this.
If you want to get a sense of how far the system can scale, you would be better with a proper performance testing framework that can run multiple threads, ideally from multiple locations (to avoid any network limits) and built-in support for accurate timing. Apache Bench is pretty common and relatively easy to setup and use. There is also JMeter and even SaaS services to do it for you.
> The problem with a lot of UI tests is that you might need to add artificial slowness to make the tests more likely to pass.
For most real-world performance tests, you should be adding plenty of delay. The average delay on the web between pages for real users runs around 50 seconds, last I looked (which was a while ago, admittedly).
If your app uses keepalives, or polling, or websockets, running your users really fast is going to make your test less accurate and you may get a false positive.
I work at Microsoft but at a completely unrelated team and while I know some of the folks who work on playwright I have no official affiliation and my opinions represent my own and not my employer's.
Playwright is great for this sort of thing! All we do is run the same test with two variants (before/after the change) and run a student t-test (well, a welch t-test but close enough).
It's ±50 LoC and as long as you're fine with running the test enough times to get statistical significance Playwright works quite well for this