[sriram_iyengar]

What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to run automation. Thoughts please.

[alixaxel]

Well, puppeteer can be used for more than test suites (think screenshots, PDF rendering, proxified APIs, ...).

But for running long automated tests, I'd probably look into alternatives like Fargate, where the billing model is per-second with a one minute minimum. Terraform + EC2 spot instances works too, obviously. :)

[sriram_iyengar]

Thanks. The example you’ve mentioned will be useful.

[tnolet]

My SaaS uses Puppeteer for website monitoring. Shameless plug: https://checklyhq.com/product/transaction-monitoring/

It's been quite difficult to get it right, with Puppeteer being young etc. but it chugs along nicely now at around 10k-15k runs per day spread over 4 regions.

[defied]

There are quite some disadvantages to use AWS Lambda to do Automated Testing (test time is capped to 15min, cold start waiting time, ...).

The advantage is that you can run a lot of tests concurrently at a relatively cheap cost.

The company [1] I work for offers VMs that are created/destroyed automatically after each test. There's no cold start, and no time limit. Plus you can choose to run headless like Puppeteer or test in an actual OS like Win/Mac.

[1] https://testingbot.com

[asien]

Lambda have a maximum running time of 15min. Even if you have « Cold Start » it won’t more than a minute for Chrome to be up and running.

Meaning you’ll get at least 10 to 14 minutes of Headless testing.

As for recommandations on how to do so , unless your testing is super long you should just run the entire thing in one function . Otherwise decouple your testing based on the various modules of your app ( i.e one module per function )

[sriram_iyengar]

Understand the 15 min and <1min coldstart. The question was more towards test suites with 100’s of tests for really large products and you cannot break them. The scenarios won’t make sense. The feedback from these tests will not be done in 15 mins.

[whoisjuan]

I think is more cost efficient and scalable than having an EC2 instance being created and destroyed every time you run a test.

[hayd]

parallelization