Hacker News new | ask | show | jobs
by Beltalowda 1482 days ago
Looking at the language, almost all of them allow you to run benchmarks since it's phrased as "you may not publish benchmark results"; it doesn't forbid to actually run them. Never mind that MS-SQL, Oracle, etc. are not SaaS vendors of course.

To be honest, if a cloud vendor has technical problems with someone running a few benchmarks then that would make me very wary of said cloud vendor. What's the difference between a "benchmark" and "using all resources I paid for" anyway?
1 comments
danielheath 1482 days ago
For a smaller/younger SAAS: If a customer environment is suddenly running at 100% of some resource when it wasn’t before, that’s an important thing to alert on / investigate.

For established players it’s lost in the noise, but if it were me I’d appreciate a heads up for big changes.

link
Beltalowda 1482 days ago
Sure, a heads-up is certainly nice, but I don't think that running a (reasonable) set of benchmarks is all that out of the ordinary, or any different from just taxing the service at 100% with some periodic batch job or the like. Paying for it is even stranger IMO.

And for what it's worth, I did actually work for a few small SaaS businesses, but a few reasonable benchmarks wouldn't have been a problem.

Of course, if your benchmarks are going to take 50 hours it's a different story.

Also: I suspect a lot of these database SaaS services are a lot smaller than you'd might think. I know at least one of them is anyway because I worked there (and there's no DeWitt Clause).

link
tetha 1481 days ago
These specific customers were not reasonable though. They wanted to specifically know when and how the software breaks, because they had been burned by previous vendors. Eventually sales reacted with a rather frustrated "then pay us 10 engineering days so we can setup a dedicated system for this so you can run your tests" and instead of cancelling the deal, they were like "Ok. Here you go, let's go"

And naturally, they were able to break it eventually, but they could have had all global employees trigger request 10 times per second and the system would've held and it recovered as soon as the load was gone. It was a silly deal, but now it's a great business partner.

link
lostcolony 1482 days ago
An occasional spike probably won't even be noticed. Redlining it when you usually hover much lower, without ceasing, -should- trigger alarms, and get engineers going "WTF is going on?!"

Give the vendor a heads up so the engineers can sleep.

link