At a glance, it appears there’s a limitation where the service runs only one test at a time. As your product and thus your testing requirements grow, this becomes infeasible. Additionally, the FAQ states that you may not get results back for 2 hours to as long as 12 hours. Both of those limitations can apparently be overcome by giving them more money for parallel tests and/or faster results.
The reason you’re running tests is to view the results and then quickly fix any bugs, re-running any failed tests to verify. Given the serial nature of testing and the turnaround time, you’re losing far more in productivity than $99/month. And if they get a sufficient number of signups at that price point, their ability to service tests is going to be severely impacted to the point where it becomes unusable.
They should really change this model and offer a limited offering for the $99/month. Keep the one-at-a-time model, but also limit the number of tests per day to something reasonable. Then introduce a $999/month plan which is unlimited. This is still way cheaper than even a part time QA tester, but as soon as you start firing sufficient load at the service you run into capacity limitations. Of course, at that point it makes sense to start offering dedicated resources for those who hit those capacity limitations, which results in a few additional offerings of $X,999 per month for various values of X. Assuming they can offer 24/7 coverage, screenshots, recordings, variables, API, etc. they’ll perhaps become a viable competitor for Rainforest QA [which I used at a previous job].
Biot you made some excellent points. The pricing at the moment is definitely exploratory. Our goal is to limit the number of people we sign up at this level and learn from the process.
I think your idea of limiting the amount of tests is great and something we will definitely explore. When we were running the service for clients manually we did something similar. As for services such as 24/7 support along with neat features like variables and interacting with API's they are on the roadmap. We are bootstrapping this at the moment and looking for our first 10 - 20 customers to work with us as we refine the service. They get a very affordable QA service and we refine our process.
Biot if you have used rainforestqa before would be great if I could ask you a couple of questions over email if possible. I can be reached at analyn (at) testrise (dot) com. Your help would be greatly appreciated
Recommend reading this re pricing: https://stripe.com/atlas/guides/saas pricing, I know you are experimenting but a single, unlimited plan is rarely a good idea.
Our main difference from rainforest is going to be that we have a completely in-house team of testers at our startup. Rainforest uses their virtual network to match your tests with appropriate testers.
The reason you’re running tests is to view the results and then quickly fix any bugs, re-running any failed tests to verify. Given the serial nature of testing and the turnaround time, you’re losing far more in productivity than $99/month. And if they get a sufficient number of signups at that price point, their ability to service tests is going to be severely impacted to the point where it becomes unusable.
They should really change this model and offer a limited offering for the $99/month. Keep the one-at-a-time model, but also limit the number of tests per day to something reasonable. Then introduce a $999/month plan which is unlimited. This is still way cheaper than even a part time QA tester, but as soon as you start firing sufficient load at the service you run into capacity limitations. Of course, at that point it makes sense to start offering dedicated resources for those who hit those capacity limitations, which results in a few additional offerings of $X,999 per month for various values of X. Assuming they can offer 24/7 coverage, screenshots, recordings, variables, API, etc. they’ll perhaps become a viable competitor for Rainforest QA [which I used at a previous job].