[latch]

This just doesn't pass the smell test. Running your own data center is a huge capital and operational investment. While there are hundreds of companies world wide that do this, they tend to be core telecommunication companies (infrastructure), dedicated data center operations, dedicate hosting companies or companies with billions in revenue.

A google search confirms that they colocate, apparently with Equinix. This is as far as running a data center as living in an apartment is to building and managing apartment buildings.

(Fun Fact, while Amazon does own and run its own data centers in most places, they're also colocating with Equinix in some regions (though I assume their level of colocation goes beyond traditional colocation)).

[kelp]

When they say "running a datacenter" they almost certainly mean "buying servers to put into rented colocation space".

Just about anyone who has significant network connectivity has a footprint in an Equinix datacenter. In the Bay Area you want to be in Equinix SV1 or SV5, at 11, and 9 Great Oaks, San Jose.

If you're there, you can order a cross connect to basically any telco you can imagine, and any other large company. You can also get on the Equinix exchange and connect to many more.

But, Equinix charges you a huge premium for this, typically 2 - 3x other providers for space and power. Also they charge about $300 per month per cross connect.

So your network backbone tends to have a POP here, and maybe you put some CDN nodes here, but you don't build out significant compute. It's too expensive.

On the cheaper, but still highish quality end you have companies like CoreSite, and I'm pretty sure AWS has an entire building leased out at the CoreSite SantaClara campus for portions of us-west-1. (Pretty sure because people are always cagey about this kind of thing.)

I also know that Oracle cloud has been well know for taking lots of retail and wholesale datacenter space from the likes of CoreSite, and Digital Reality Trust, because it was faster to get to market. This is compared to purpose build datacenters, which is what the larger players typically do.

In the case of AWS, I know they generally do a leaseback, where they contract with another company who owns the building shell, and then AWS brings in all their own equipment.

But all these players are also going to have some footprint in various retail datacenters like Equinix and CoreSite for the connectivity, and some extra capacity.

Zoom is probably doing a mix of various colocation providers, and just getting the best deal / quality for the given local market they want to have a PoP in. Seems like they are also making Oracle Cloud part of that story.

[KorematsuFred]

So many people forget that running data center is a super complex business not just in point of technology but also in terms of operation.

I have known people who tried to setup a data center in India and it took them around 2 years to have the first rack installed. Biggest hurdle was to get a license to store fuel in large tanks for their generators. Not to mention many of those permissions have to be renewed annually and if you fail to renew it which can take months, you are not in compliance and hence can't use the generators.

In India you can not start your own power generation plant and you can sell electricity only to the government. Depending on many situations you have to technically register a separate entity, get licenses as a "power company" then on paper sell the electricity to government and then buy it back from government for your own use.

[mmmBacon]

It’s difficult but it’s not that hard and it’s well worth the investment. When I was flirting with founding a company I looked at this trade off and with hiring/overhead etc... it was significantly cheaper to roll our own than it was to use AWS. It was only cheaper at the earliest stage.

The only actual reason to use AWS is to not divert any energy to doing anything else but scaling the company. The only problem was that by the time you are at some reasonable scale, AWS has you pretty locked in.

[blueblisters]

You can get around some of these operational constraints with technology. For example, Google had a server design with its own in-built backup battery supply, which incidentally could be cheaper than diesel generators. So backup power for your servers is solved but you still might need to figure out backup power for other parts of the datacenter.

[freepor]

Batteries don’t have anywhere near the energy density of hydrocarbons. Batteries are good for a few hours, but if you want to be able to run for days off-grid you will need hydrocarbons.

[throwaway888abc]

This, The certification barrier is steep and clients wants reliable DC (with ISO, redundant and working power, connectivity, etc.).

[kartickv]

At what scale does colocation make sense as opposed to a long-term commitment on AWS or other clouds, where you get a discount?

[TechBro8615]

If you’re building an infrastructure company, or any bandwidth intensive product, it makes sense pretty early. For example, it would be impossible to build a competitive CDN or VPN company based on AWS infrastructure. It would also be hard for Zoom to offer a free tier if they were paying per gigabyte for bandwidth, since it would essentially mean every extra second of a meeting cost them money directly.

The fundamental problem is that AWS (or any major cloud) charges you for the amount of “stuff” you put through the pipe ($/gb), but with colocation you can pay a fixed cost for the size of the pipe ($/gbps). This allows you to do your own traffic shaping and absorb bandwidth costs without needing to pass them onto your customers.

This is the dirty, open secret of cloud pricing models. It’s also their moat, which makes it infeasible to do something like “build AWS on AWS.”

[kelp]

Yeah, this is really where they get you.

For context, if you were to buy 10Gbps of dedicated internet transit, he.net is currently advertising that for $900/month.

If we convert that to GB per month it's 3,240,000GB, so we can calculate what AWS would charge based on list prices.

Using their pricing calculator: https://calculator.aws/#/createCalculator

Outbound from Cloud Front or US West (Oregon) to the Internet:

$165,891.11

That's a 184 x increase in price!

So yeah, you have to buy networking gear and other stuff, but you can get quite a bit of gear for $165K/month. Now you don't really want to run that 10Gbps link flat out like that, but you get the point.

The AWS markup on bandwidth costs is absolutely insane.

Pro tip: if you have a large enough cloud provider spend, you can negotiate the bandwidth prices down quite a lot, given their markup, they have some room to move.

[kitteh]

In China the cloud provider pricing model is that you have a slider bar to select how many Mbps your instance can scale up (maxes at 100 to 200 Mbps). But that has more to do with controlling customers ability to burst whereas most other providers do the GB xferd. Some of the China provides have adopted the GB xferd recently, though.

[kartickv]

So the way the slider works is that you pay for the max mbps irrespective of data transferred?

[donavanm]

My swag would be around the $500K/mo opex and $5-10M/year capital expenditure mark it’s worth a conversation. I haven’t been deeply involved in infrastructure for a few years now. I did work in compute farms, networking, CDNs, etc for about 15 years previously. In my old comments you can find more detailed math on swagging out actual network & infra costs for “equivalent to cloud” infrastructure.

[kelp]

This sounds about right to me. The last time I was doing that kind of work it was at slightly large scale than that, and it always modeled out cheaper to stay with our on gear, in colocation space.

[rstuart4133]

Any scale where you get good utilisation of the metal. You can rent a dedicated 8 core server for $100/mo or so, 1Gbps unlimited usage. If you use it at full throttle it is literally under 1/10'th the price, often under 100'th of the amount you will pay for a cloud host and the thousand little cost addon's.

But the trick is you have to actually use it and need it in real time. An AWS instance costs you nothing if you don't use it, and almost nothing if you let them kill it at their whim.

Zoom's strategy looks pretty optimal to me. Take the 100 fold price reduction on your predicable load, farm the rest out to the lowest bidder.

[barkingcat]

Having servers in a colo is pretty much what "having a datacenter" means for the last 10 years or so. I'm surprised at your viewpoint of "having a datacentre" meaning only having direct ownership of the piece of land on which the datacentre resides. That's a pretty rigid and outdated definition.

[oneplane]

Maybe where you live, but not where I live and probably also not where he lives.

When we (locally) talk about having your own hardware we use terms like 'bare metal' and 'machines' or even the specific names of the types of machines, like 'the blades' or 'the dells'. We really only use 'datacenter' when we talk about a physical location we own, with power we own, cooling we own, networking we own, and access control we own. Otherwise it's just colo.

[jayp]

Where is this place? I agree with grandparent - for the past decade or so, having your “own datacenter” means leasing floorspace in an actual datacenter run and managed by someone else.

I live and work in the Bay area. I was part of the larger org that ran Uber’s datacenter (co-located) and that’s how it was talked about.

[latch]

If 'having your "own datacenter"' means colocation, how do you say "having your own datacenter"?

[jayp]

Good question. Probably the same. English, and especially business English, is not a precise language. I am not really an expert but was part of the larger org.

Just to be clear - colocation here is in order of thousands of square feet. The datacenter provider provides redundant utilities. The customer does everything else.

[BubRoss]

Seems a little silly to blame english when people bend terminology to muddy the waters on purpose. When 'serverless' means easily using a server and 'we have our own datacenter' means 'we don't have our own data center and rent from someone else' maybe the problem is people trying to stretch the significance of what they are doing.

If someone says 'I own my house' and they actually mean 'I rent an apartment', no one is going to say 'oh well, english is an imprecise language'.

[oneplane]

Western Europe and Scandinavia mostly.

[jcims]

I would imagine most of that AWS colo is for hybrid cloud products (Direct Connect/Outposts/etc.). It’s obviously possible but seems unlikely they would run an availability zone in a space unless they had complete and exclusive control over the physical plant.

[jlawer]

They use exclusive sections of an existing colo provider facilities, often called data suites or data halls at least in Australia, where the 3 AZs are split between numerous commercial colo facilities.

They get enough control through their contracts to make sure the hosting provider provides exactly what they need to spec.

[kelp]

Yeah, this is correct. You usually buy these in minimum sizes of about 1MW.

[jcims]

Interesting! Thanks for the info!

[kelp]

Yeah, you can see the Direct Connect locations here: https://aws.amazon.com/directconnect/features/

Those mean AWS has networking gear in those locations. You order a cross connect and plug into one of their switches.

When you walk around in one of those places, you also typically see racks of AWS gear in a smallish cage with lots of hard drives. Typically a CDN pop.

[dmw_ng]

There was some leaked corp doc from Amazon (Wikileaks maybe?) that showed significant chunk of their DC space had a recognizable name attached to it, it looked a lot like they often just buy this stuff in too.

[skuhn]

Yes, Amazon buys space just like everyone else. I have been in lots of buildings or campuses where they were located.

They buy a lot of space, and they work with lots of providers, but they don't own very many sites. So I guess they don't "have a datacenter" for the purposes of this thread. No one I know in the business thinks this way in the year 2020.

Designing and operating datacenter facilities is specialized work, and it's about compliance, auditing, risk management, electrical, plumbing, hvac and other skilled trades. The datacenter industry actually has very little to do with computers, so there is a natural split between the facility and the server / network equipment it houses.

Basically all commercial datacenter providers operate as REITs, which is tax advantageous but extremely limiting in some ways. Amazon can benefit from this (with lower pricing) without dealing with it themselves.

Owning can offer some advantages, but it also means you're with that site for the long, long haul. Efficiencies of designs are always increasing, so operating in an old facility costs you money. If you built the site to your own spec, good luck exiting -- the next owner will have to do a total overhaul to get it to industry spec and get customers.

Even if you have a 10 year lease, there are always ways to get out if you want to. Especially if you're Amazon.

[cobookman]

Google & Facebook both build their own Datacenter facilities.

But they operate at a scale that is very unique.

[skuhn]

They build sites, but they also lease space from the same providers as everyone else.

It's also worth noting that sometimes when a company builds a datacenter in a green field situation, it may be working with a datacenter provider on that project. So the company may own it, but they're paying the provider to use their design elements and potentially to operate it.

[Spooky23]

For a big company? Not really.

I’ve done big datacenter builds and cloud projects. The cloud projects were usually not a big win from a cost POV in my experience. Financially it’s a tax and marginal unit of capacity play.

Definitely easier to manage at a certain level, especially with variable demand or to accommodate growth.

[kitteh]

Amazon has to be in colos for their network infrastructure to interconnect with their peers (private peering and internet exchanges) and transit providers. Direct Connect also in there for similar reasons (to be closer to customers). This is all public information and peeringdb.com shows what locations they're in.

[rantwasp]

yeah no. Amazon does have data centers. A lot. Maybe you’re talking about points of presence for something like CloudFront where indeed it’s not feasible to build a datacenter in each location - but for major regions AWS has its own datacenters.