[tetha]

Interesting. As a SaaS vendor, we do not allow performance testing of the production system. Because, you know, just casually saturating production resources can become very iffy for strange and unexpected reasons. And you will always be able to saturate a system, or a subsystem of the subsystem of the system.

However, we have provided bigger customers, or customer willing to pay for it, with performance testing environments. We have, however, usually survived into the curiosity phase - "just how much to I have to throw at this thing to break it?".

[Beltalowda]

Looking at the language, almost all of them allow you to run benchmarks since it's phrased as "you may not publish benchmark results"; it doesn't forbid to actually run them. Never mind that MS-SQL, Oracle, etc. are not SaaS vendors of course.

To be honest, if a cloud vendor has technical problems with someone running a few benchmarks then that would make me very wary of said cloud vendor. What's the difference between a "benchmark" and "using all resources I paid for" anyway?

[danielheath]

For a smaller/younger SAAS: If a customer environment is suddenly running at 100% of some resource when it wasn’t before, that’s an important thing to alert on / investigate.

For established players it’s lost in the noise, but if it were me I’d appreciate a heads up for big changes.

[Beltalowda]

Sure, a heads-up is certainly nice, but I don't think that running a (reasonable) set of benchmarks is all that out of the ordinary, or any different from just taxing the service at 100% with some periodic batch job or the like. Paying for it is even stranger IMO.

And for what it's worth, I did actually work for a few small SaaS businesses, but a few reasonable benchmarks wouldn't have been a problem.

Of course, if your benchmarks are going to take 50 hours it's a different story.

Also: I suspect a lot of these database SaaS services are a lot smaller than you'd might think. I know at least one of them is anyway because I worked there (and there's no DeWitt Clause).

[tetha]

These specific customers were not reasonable though. They wanted to specifically know when and how the software breaks, because they had been burned by previous vendors. Eventually sales reacted with a rather frustrated "then pay us 10 engineering days so we can setup a dedicated system for this so you can run your tests" and instead of cancelling the deal, they were like "Ok. Here you go, let's go"

And naturally, they were able to break it eventually, but they could have had all global employees trigger request 10 times per second and the system would've held and it recovered as soon as the load was gone. It was a silly deal, but now it's a great business partner.

[lostcolony]

An occasional spike probably won't even be noticed. Redlining it when you usually hover much lower, without ceasing, -should- trigger alarms, and get engineers going "WTF is going on?!"

Give the vendor a heads up so the engineers can sleep.

[josephcsible]

There's a really big difference between "don't performance test on our hardware that you're sharing with other tenants" and "don't performance test on our software no matter whose hardware it's running on".