[eldavido]

This is terrible advice for all but the largest of organizations.

Running your own hardware is AWFUL. Get ready to dedicate an entire team to network engineering, fixing broken hard disks, patching operating systems, screwing around with RAID controllers, upgrading switches, planning power and cooling, and endless vendor negotiation -- with ISPs, hardware manufacturers, datacenter operators, etc.

Oh, and did I mention, throw elasticity out the window -- all of this has to be planned in advance, and purchased, and installed, months ahead of when it will be operable. So forget the ease and convenience of just spinning up more capacity.

Also, there's a massive distraction of having to focus management attention on this non-value-adding part of the business, all so that you can shave down some cost, rather than investing in growing revenue.

Having been in this position firsthand, don't do this. If netflix can run 1/3 of Internet traffic off of AWS, I guarantee they're much larger than you, and it should say something, that they'd rather outsource this part of their business than dealing with all this crap.

Focus on software and product/market fit. It's just a much, much, much better use of expensive technical people, that will be done on day 1, without any risk, hassle, or complexity, than trying to replicate something someone else already does for you, much better and at competitive costs, than trying to reinvent all of this yourself.

To be honest, I don't even want to deal with EC2 anymore; I'd rather just use a PaaS.

[hueving]

>This is terrible advice for all but the largest of organizations.

Don't start a conversation with an opening generalization like that if you want something constructive. Especially when the rest of your post is clearly based on the single anecdote of your experience.

>Running your own hardware is AWFUL.

Maybe for you. Not for any sysadmin with even just a couple of years of experience.

>patching operating systems

We're talking about instances, none of that sysadmin stuff goes away if you're on AWS. If you don't have patching management for operating systems on AWS then your instances are screwed. AWS instances don't eliminate the need for sysadmin work.

The only real difference is the hardware management. And if you read my post you would have seen that I said using aws for the on-demand flexibility is okay. All of the static workloads are what belongs down in your datacenter.

Netflix doesn't run 1/3 of the Internet traffic off of AWS, only a tiny subset because of the aforementioned shitty economics. The real workhorses are in custom netflix servers at peering points. Netflix would be bankrupt if they used AWS for video. Do some research before spreading free marketing propaganda.

This forum tends to only think in terms of explosive growth of traffic, which <0.1% of companies actually have to deal with. AWS flexibility is needed by very few successful B2C companies, but it's supported by the cargo-culting of orders of magnitude more developers ("jedberg said this worked for reddit, we need this because we're like reddit").

Also, your whole argument about non 'value-add' is bogus. That's the same excuse that management uses to outsource all development. Everything has a cost and provides some value to the company.

[not_kurt_godel]

> Don't start a conversation with an opening generalization like that if you want something constructive. Especially when the rest of your post is clearly based on the single anecdote of your experience.

You are doing as much, if not more generalization by way of the assumptions you're making.

> >Running your own hardware is AWFUL. > Maybe for you. Not for any sysadmin with even just a couple of years of experience.

Sysadmins aren't real estate attorneys or facilities managers or security personnel. A lot of them aren't even tech ops who physically manage the DC hardware and are oncall 24/7 to fix problems on-site if needed. You need all of those things to run your own DC, and potentially a lot more if you're physically building the center itself (architects, contractors, civil engineers, etc.). And if you're serious about latency, availability, and durability, you're going to need those things in multiples for however many datacenters are needed to meet your targets. How many organizations have the millions of dollars of capex needed to get that off the ground and keep it all running?

> Netflix doesn't run 1/3 of the Internet traffic off of AWS, only a tiny subset because of the aforementioned shitty economics.

To how many organizations do the economics of serving 1/3 of Internet traffic apply? 2? How is that a counterexample to his point about datacenters only making sense for the very largest?

Even if you sidestep all those costs by renting instead of building and even if we take for granted that your "shitty economics" are still shitty down numerous orders of magnitude from Netflix-size, you're still burning money making your devs design and operate your system twice - once for the DC, once for AWS, and however much work it is to glue the 2 together. The end result may be cheaper infrastructure-wise but it will also be unavoidably less reliable and more complex purely by virtue of having more than twice as many moving parts.

Let's say implementing things this way takes a single dev time-and-a-half compared to just doing it on one or the other. Let's say (very conservatively) you pay your dev $100k a year + $50k benefits. You're now $75k in the hole from the get-go. That's enough to pay for roughly 83 m5.large EC2 instances on-demand (no RI) for a year. Your company has to be very large for the marginal savings of using a DC to outweigh that kind of deficit.

[kakwa_]

> How many organizations have the millions of dollars of capex needed to get that off the ground and keep it all running?

I think you are exaggerating when talking about "millions". Companies running in old school DC will not build the actual DC. They will rent a 42U cabinet or part of it.

You can typically rent 1/4 of a cabinet for a few hundred bucks a month.

I you operate at a small scale, you typically don't have that many servers, maybe 4 or 5. A decent server is in the range of 4000 to 6000$ and will last generally around 5 years.

And these servers rarely breaks (we have a fleet of more 300 servers, and maybe 1 or 2 "wake the on call" crashes a year).

Throw in 1 or 2 decent switches at about 1000 to 2000$ each and you are mostly good to go.

You end up with a capex of ~50 000$ to get you started, with a depreciation over 5 years.

[eldavido]

My experience was a little different.

Power is the dominant cost in datacenters unless you have very, very expensive switches. Power drives direct electricity consumption, the need for backup batteries and UPSes, cooling, and fans.

You can get pretty high-powered supermicro machines for only like 1000 or 2000 dollars these days. Over 5 years, that works out to only $16/month.

A 42U cabinet from HE (Hurricane Electric) that's "on sale" (http://he.net/colocation.html) runs $400/month. You need 20-30 servers before the spend on machines starts to overtake power. And I honestly doubt you'd be able to put 30 machines in that cabinet before hitting their power ceiling. I walked through an Equinix facility a while ago and if you hit 10KW/rack that's considered "hot". It's not hard to do if you stuff an entire 42U rack with 1U multi-core machines that each have CPUs drawing 50-100 watts/core (not unlikely with high-end Xeons). 15KW/rack is really hot.

[not_kurt_godel]

What kind of specs are you getting for that kind of money? Just played around with AWS pricing calculator and honestly it seems like EC2 is even more cost-effective than I thought, depending on specs...

Edit: Try using https://awstcocalculator.com and see what it claims your savings would be (I'm curious how accurate it is)

[kakwa_]

Doing a quick check with Dell customization. For 6000$ I can get:

CPU: AMD EPYC 7351P (AMD EPYC™ 7351P 2.4GHz/2.9GHz, 16C/32T,

64M Cache (155W/170W) DDR4-2400/2666)

RAM: 64GB

Storage: 2x 800GB NVMe drives

+ dual power suply + Rails with cable management + 5 years support

Over 5 years, it's 6000 / 12 * 5 = 100$ per month.

Something comparable like an i3.2xlarge (less CPUs but more storage) is at 455.52$ per month On Demand price.

3 years full upfront, it's at 192.72$, better but still more expensive.

And i3 are less convenient than your server because they can go down at anytime. This means loosing the local NVMe storage you have to come up with a mix of EBS volume+local NVMe if you want some persistencies, or heavy clustering where loosing a node is not a big deal.

AWS is really expensive if you are using it wrong. And a lot of us are using it wrong ('"move <INSERT LEGACY APP> to the cloud" says management' mode). Using it right is quite difficult in fact, it requires a lot of engineering complexity to be resilient when AWS chose to shot the hypervisor under your feet. And truly leveraging the elasticity provided by things like AutoScaling groups or Lambdas, specially at the storage layer, is far from simple. I've seen instance where attempts to build "SERVICE THAT SCALE" ended-up being even worst than <LEGACY APP IN THE CLOUD> in term of costs.

This has been my experience being part of a team managing a large mixed deployment with ~5000 EC2 instances on one side, and, on the other 300 physical servers, each handling ~10 LXC containers in legacy DCs.

Where AWS shines is the fact you get a lot of flexibility. Need more capacity? increase the instance size and you are good to go. Need to create 300 ELB in a rush with some DNS records? it's done in 1 or 2 hours and 80 lines of boto. That level of instrumentation is not maintainable by any companies except Cloud providers and the biggest internet actors.

But if your load is fairly static and/or predictable, if you have the capital to buy the servers upfront, if your customer base is fairly localized, if you can manage the added complexity of having to do some capacity planing and hardware inventorying once or twice a year, then legacy DCs are still cheaper but I know and understand it's a lot of IFs.

[apple4ever]

That has exactly been our experience when deciding to go AWS vs self hosted. For our environment, it makes sense for time and money to go self hosted. I get it may be different than others, but not for use.

[dominias]

You got to accept that building and maintaining data enters are simply an old and obsolete model of doing IT that will eventually go away.

[niftich]

I get what you're saying about the benefits of PaaS or other managed services, but Netflix serves most content (by volume) off of appliances [1] they ship to ISPs that feed into the network at points-of-presence somewhere close to an edge. They use AWS for logic and compute (including transcoding [3]), but video streams from this globally-distributed CDN [2].

[1] https://openconnect.netflix.com/en/ [2] https://media.netflix.com/en/company-blog/how-netflix-works-... [3] https://medium.com/netflix-techblog/high-quality-video-encod...

[patrickg_zill]

Having run a small hosting company for about 17 years now, I am laughing at this post... You greatly overstated the difficulty and understate the massive cost savings.

[darkerside]

When it's your business, of course these things are easy. It's just part of routine maintenance and overhead. When you're in a different business, you generally want to be focusing on your own core competencies and solving your own specific business challenges, not care and feeding of servers. Classic division of labor problem.

[outworlder]

No they didn't. Because you are a hosting company. Not all companies have your expertise - most don't.

[sitkack]

The hyperbolic difficulty that folks are applying to running your own hardware is laughable. It is as if by miracle alone that civilization made it through the precarious transitionary period where we racked our own servers into the Wonderous Utopia of Cloud.

I definitely wouldn't tie my own shoe laces, only the largest orgs need laces, almost everyone can focus on walking if they use Velcro.

[tatersolid]

It’s about staff and focus.

Right now we have 3 sysadmins for our own DC, and they can’t keep up with all the maintenance (upgrades on infra are the worst. Software quality on switches/SANs/appliances is terrifying. And those are better than quality of Server management, BIOS, etc. Even VMware has caused lots of pain with patches and upgrades in recent years.

If you don’t ever patch/upgrade to fix all the security vulnerabilities that exist at the infrastructure layer maybe “do it yourself” is cheaper. But that doesn’t fly in the banking industry.

We’re half-in the cloud and moving more there just to keep our head above water. We’re spending way less on cloud services than the $500K/year it would cost to double our infrastructure staff and keep it all in house.

[eldavido]

I've written a lot on this thread. I think the linchpin of this entire thing is how much salaries for technical people have lifted.

If you can get a qualified dev/dc engineer for 50 or 60K maybe you rack your own. But when facebook is paying all-in median wages of 240K/year things start to look different. I understand not everyone is going to be able to work at fb and not everyone lives in California. But when one of your critical inputs doubles or triples in price, of course that's going to adjust how things get done an in industry.

[Johnny555]

It's been a few years since doing the cost benefit analysis between AWS and self-hosting, but for around 50 racks worth of servers and storage, the numbers came in on AWS's side.

That didn't even take into account the "free" multi-region capability you get from Amazon. Splitting our physical servers into a second region with enough capacity to failover would have nearly doubled our costs.

[hueving]

Were those numbers using 50 racks worth of instances (e.g. 20,000 of them) for the comparison? Did you remember to take into account the obscene bandwidth rates (sorry if that seems like a dumb question but I've seen this bite multiple companies moving to AWS)?

The break evens happen a lot earlier in my experience for static workloads, but I would love to see a breakdown if you're willing to share details.

[jjeaff]

Why would you compare AWS vs managing your own data center?

You could also compare AWS vs building your own silicon.

I think it would be better to compare AWS vs renting dedicated servers from a large provider? I think you will find that the scales tip heavily in favor of renting bare metal as far as price is concerned.

[Johnny555]

Why would you compare AWS vs managing your own data center?

Because we were already managing our own data center.

I think it would be better to compare AWS vs renting dedicated servers from a large provider? I think you will find that the scales tip heavily in favor of renting bare metal as far as price is concerned.

We offloaded a lot of work to Amazon that we were doing ourselves -- database hosting, storage system management, etc (lots of little used data went into S3/Glacier that previously we had on live disks)

Also, we liked the ability to have a failover region essentially for free - we only pay for enough servers to replicate the key data we need for failover, and keep the rest of the infrastructure powered off.

[mmt]

> storage system management

I was a bit incredulous that any truly all-inclusive analysis could ever show AWS being cheaper, but this phrasing made me realize that it could have been the one (remarkably common) case where it usually does: enterprise hardware.

That world is easily more expensive than AWS, especially considering that hardware maintenance contracts are a thing (and a shockingly expensive one, to those of us accustomed to the commodity hardware world).

> Also, we liked the ability to have a failover region essentially for free - we only pay for enough servers to replicate the key data we need for failover, and keep the rest of the infrastructure powered off.

That's a useful advantage, though there's a pitfall in that there's no powering off EBS volumes.

[dominias]

You may be managing your own datacenter but you sure won’t be building any new ones.

[apple4ever]

I think that would make sense. For us, that need a single rack, it comes out heavily on the data center size. AWS was twice as expensive.

[toomuchtodo]

Netflix runs a third of internet traffic off of their OpenConnect CDN appliances, not AWS (Netflix.com and other control plane-ish functions aside).

Once you’re spending a few million dollars a year on cloud (arguably even less in some cases), it behooves you to explore hosting yourself. Cloud provider margins are substantial for a reason.

[eldavido]

If you're truly spending "a few million dollars"/year, then MAYBE.

200k/month would get you there. My personal breakpoints are 50k/mo PaaS->EC2/IaaS and then something like 100k or 150k start thinking about physical. Maybe. By the time you hire everyone, get all the planning right, etc. you might be there need-wise, but it's not a sure thing.

[jjeaff]

Why not just rent dedicated bare metal from a large provider?

[mmt]

I'd expect that to have nearly all the disadvantages of cloud (other than virtualization).

One is still locked into the provider's pricing structure.

One is still locked into the provider's ISP choices.

One is still locked into the provider's internal network architecture choices.

One is still locked into the provider's limited hardware choices.

The last one is the biggest one, if only because there are so many opportunities in so many components to maximize performance and minimize cost, with a little forethought and customization, all while staying well within the commodity market. The "one size fits all" model does many people a disservice.

This is especially true even at fairly low scale, where the issue of obtaining certain components in large volumes doesn't come into play.

I understand why some people insist it's just too hard to deal with hardware, but I find it disingenuous to advocate that opnion without at least a more comprehensive understanding of what modern, commodity hardware is capable of. I think most of the claims of difficulty are coming from those without this knowledge and have primarily software backgrounds (and are relying on hearsay or other second-hand experience).

[jjeaff]

Most providers I have seen that rent hardware offer unlimited customization when it come to the box you are renting. Network architecture would obviously be off the table though.