What about the personnel costs? Also, what about the instantaneous bandwidth. If you have huge demand spikes where you need a lot of bandwidth at one time, can your system handle it?
"What if we grow 20x overnight" is basically magical thinking, that's about what it'd take to really cause problems, the odds of it occurring are dirt low. Most of the time it'll lead you to waste money. Certainly not enough to account for nearly that price increase. Especially given that you can still rent dedicated or cloud servers temporarily to accommodate in the event that something like that does happen.
As for personnel, if I had to, I'd hire mostly devops people or pay freelancers for jobs as needed until hitting a limit where you have to dedicate a large portion of someone's time to it and then repeat the math - I fully suspect that this would mean sticking with bare metal. I suspect a 2 or 3 person group dedicating their time to managing only hardware and basic infrastructure for something like Kubernetes could probably handle hundreds of servers without issue. Even if that averages out to an additional cost of $1000 per server per year it'd only be approximately a $80 per server month, nothing like the increase of going to EC2. And renting entire racks gets cheaper than the individual colocation I'm paying for at the moment. It'd surely be interesting
I have no idea where people came from when they moved to AWS/etc but I always have (and still do) picked up the phone and got someone on the line who usually fixes whatever the issue is with me on the phone.
As far as demand spikes - even smaller hosting companies will have hardware they can spin up pretty quick.
As for personnel, if I had to, I'd hire mostly devops people or pay freelancers for jobs as needed until hitting a limit where you have to dedicate a large portion of someone's time to it and then repeat the math - I fully suspect that this would mean sticking with bare metal. I suspect a 2 or 3 person group dedicating their time to managing only hardware and basic infrastructure for something like Kubernetes could probably handle hundreds of servers without issue. Even if that averages out to an additional cost of $1000 per server per year it'd only be approximately a $80 per server month, nothing like the increase of going to EC2. And renting entire racks gets cheaper than the individual colocation I'm paying for at the moment. It'd surely be interesting