[toomuchtodo]

Netflix runs a third of internet traffic off of their OpenConnect CDN appliances, not AWS (Netflix.com and other control plane-ish functions aside).

Once you’re spending a few million dollars a year on cloud (arguably even less in some cases), it behooves you to explore hosting yourself. Cloud provider margins are substantial for a reason.

[eldavido]

If you're truly spending "a few million dollars"/year, then MAYBE.

200k/month would get you there. My personal breakpoints are 50k/mo PaaS->EC2/IaaS and then something like 100k or 150k start thinking about physical. Maybe. By the time you hire everyone, get all the planning right, etc. you might be there need-wise, but it's not a sure thing.

[jjeaff]

Why not just rent dedicated bare metal from a large provider?

[mmt]

I'd expect that to have nearly all the disadvantages of cloud (other than virtualization).

One is still locked into the provider's pricing structure.

One is still locked into the provider's ISP choices.

One is still locked into the provider's internal network architecture choices.

One is still locked into the provider's limited hardware choices.

The last one is the biggest one, if only because there are so many opportunities in so many components to maximize performance and minimize cost, with a little forethought and customization, all while staying well within the commodity market. The "one size fits all" model does many people a disservice.

This is especially true even at fairly low scale, where the issue of obtaining certain components in large volumes doesn't come into play.

I understand why some people insist it's just too hard to deal with hardware, but I find it disingenuous to advocate that opnion without at least a more comprehensive understanding of what modern, commodity hardware is capable of. I think most of the claims of difficulty are coming from those without this knowledge and have primarily software backgrounds (and are relying on hearsay or other second-hand experience).

[jjeaff]

Most providers I have seen that rent hardware offer unlimited customization when it come to the box you are renting. Network architecture would obviously be off the table though.