[gruez]

Is there a plausible explanation why egress fees from cloud providers costs around $0.1/GB? "Traditional" server providers such as Hetzner are able to offer bandwidth at orders of magnitude lower price (eg. $1.1/TB). I understand that cloud providers may have better interconnects or better uptimes, but that doesn't justify the magnitudes higher pricing.

[ben509]

Disclaimer: I worked on an AWS service team.

This is, oddly enough, similar to a debate people have about consumers TV or Internet: should pricing be "unlimited" or "a la carte"?

AWS is combining all your networking charges into one lump "outgoing data transfer" fee. So it's heavily marked up in comparison to what they're paying for the outgoing data transfer, and you're not sure how much is profit vs. whether it's going to cover all their other costs.

So it might be fairer if AWS broke out separate line items for internal, incoming and outgoing data transfer, plus all the additional systems a customer uses.

I think AWS's billing is probably already on the falling side of diminishing marginal returns. That is, it's complex enough that more information would tend to hinder customers from getting the best price. Right now, if I plan to reduce my data charges, I have one variable to tinker with. If we expand this, it would mean I'm having to balance incoming / internal and outgoing charges. That sounds simple, but in terms of engineering it can be very complex.

The next claim is that this biases customers not to move. Of course, Azure and GCP have the same arrangement, so while you pay to move out of AWS, you don't pay to move in to Azure or GCP. So all the vendors are attempting to lock you in to their product, and at the same time trying to extricate you from their competitors, overall it's a wash.

So, yes, part of the motivation for egress charges is that ingress is a loss leader. But it's also true that egress is a metric that does, for the vast majority of their customers, directly translate into customer value. If there's a compelling case for doing it differently, someone should do it and see if it works.

[throwaway_bad]

> If there's a compelling case for doing it differently, someone should do it and see if it works.

Cloudflare doesn't charge for bandwidth. I always throw cloudflare on top of anything I do, not because I really need a CDN or anything, but because the bandwidth cost would bankrupt me otherwise. The ceo of cloudflare gave the rationale on why they don't charge:

> There’s a fixed cost of setting up those peering arrangements, but, once in place, there’s no incremental cost. That’s why we have similar agreements to Backblaze in place with Google, Microsoft, IBM, Digital Ocean, etc. It’s pretty shameful, actually, that AWS has so far refused. When using Cloudflare, they don’t pay for the bandwidth, and we don’t pay for the Bandwidth, so why are customers paying for the bandwidth. Amazon pretends to be customer-focused. This is a clear example where they’re not.

https://news.ycombinator.com/item?id=20791563

[qes]

According to Cloudflare, they do not have any bandwidth pricing arrangement with Microsoft for Azure users.

They also do charge for Enterprise plans, but instead of transparent pricing I got high-pressure sales techniques and black box pricing offers - which then anchored our rate so that as we grow past our current contract, we're forced to upgrade at any point with pricing based solely on our original negotiation.

Frankly, while I save money using Cloudflare over Azure's CDN right now, it's left a very sour taste in my mouth and I'll be jumping their ship as soon as I have time to find a suitable alternative.

[bam_boo]

> high-pressure sales techniques and black box pricing offers - which then anchored our rate

If you have the ability to shift your entire enterprise CDN away from them, why not first try renegotiating?

[dberg]

Just FYI in case it helps you - https://docs.fastly.com/en/guides/outbound-data-transfer-fro...

[austinheap]

Cloudflare most certainly disables zones on the free plan that use excess bandwidth. Enterprise contracts are also negotiated based on transit and those prices mirror comparable CDN services.

[bmurphy1976]

All that tells us is that cloudflare has a different revenue stream. Amazon is a business and they are in the business of making money. If they weren't charging for egress bandwidth they'd just charge for something else.

[rsync]

"I think AWS's billing is probably already on the falling side of diminishing marginal returns. That is, it's complex enough that more information would tend to hinder customers from getting the best price. Right now, if I plan to reduce my data charges, I have one variable to tinker with."

No, it's two variables - the egress charges you refer to and the actual cost to store the data.

We[1] have found that it is, as you might expect, quite a bit simpler to charge for just the storage and forget about metering the usage/bandwidth/transfer.

So we have typically had our price point higher than the B2s or Wasabis of the world, but there's just one simple number to think about - and no potential for surprises in the billing.

I will admit to having a bit of concern over adding 'rclone'[2] to our platform and the potential for users to just burn bandwidth using an rsync.net account as a "transfer host" but that is why we peer with he.net and their cheap an plentiful 10gb pipes.

[1] rsync.net

[2] ssh [e-mail] rclone s3:/bucket gdrive:/blah/blah

[nostrebored]

And how many PoPs regional interconnects, highly availabile, high throughout connections, cross continent highly available connections do you have? Do you detect failure across these connections? Do you detect grey failures? Do you have a team of infrastructure engineers to look after this network?

[rsync]

We keep all of those to an absolute minimum and avoid as much complexity in our infrastructure as possible.

Which is to say, each of our five[1] regional POPs have a single connection provided through a dumb switch one hop from he.net[2].

They have no interconnection or dependencies to one another.

No routers, no firewalls, no balancing, no failover. When rsync.net fails, it's a very, very boring failure.

We've had zero network outages in the last 60 months or so.

[1] Fremont, San Diego, Denver, Zurich, Hong Kong

[2] init7 in Zurich ...

[tgsovlerkhgsel]

> So it might be fairer if AWS broke out separate line items for internal, incoming and outgoing data transfer, plus all the additional systems a customer uses.

The example given above for comparison, Hetzner, also doesn't charge for inbound and internal transfer AFAIK. Nor "the additional systems a customer uses". You pay a charge for the server, you get some amount of traffic included, and if you go over, the additional traffic costs something like $1.1/TB. That's all you pay.

[GordonS]

> AWS is combining all your networking charges into one lump "outgoing data transfer" fee

> So it might be fairer if AWS broke out separate line items for internal, incoming and outgoing data transfer

This explanation doesn't cut it for me - most (all?) "traditional" VPS providers don't charge for ingress traffic, and I doubt anyone, ever, has charged for internal traffic.

So what exactly is 'all the networking charges' comprised of, other than egress data?

[ben509]

AWS is vast. I have no idea what their overall accounting for networking looks like. Even for the tiny service I worked on, it would be tough to guess at what are overall costs were. We actually had an internal bill each month for all the regular AWS services we used, but then there were a host of internal services we depended on.

That companies don't charge for specific things doesn't mean those things don't cost them anything. It just means they're trying to work out a pricing scheme that scales with customer usage and is broadly understandable. So "data egress" is really just a proxy for "how much stuff you're doing with the networking subsystems of AWS."

Same thing with EC2, there are a whole pile of costs that are summed up with "time you rented an instance."

[GordonS]

See a lower comment I made here; what I really want is a little transparency about pricing.

Of course there are is an internal cost of doing business, and peripheral infrastructure cost - but if I pay $100 for service "A" I reasonably expect that fee pays for service "A". Instead, egress bandwidth costs seem to be used to trick customers into thinking services are cheaper than they really are.

[nostrebored]

How many of these VPS providers are actually managing global highly availabile network infrastructure?

[GordonS]

I would have presumed that the infrastructure cost for each service was included in the cost for each service.

For egress bandwidth costs, I'd assume it included, well, the egress bandwidth cost.

[e12e]

I guess there aren't that many global network providers - I'm not even sure how much fiber Amazon owns in Japan, Australia or Northern Europe for that matter.

But I think level3 is associated with:

https://www.centurylink.com/business/hybrid-it-cloud/public-...

And while they have a call-us price list (if you have to ask...) - they at least state:

"Public and Private high-capacity networking options up to 10Gbps. Note: there is no charge for internal data center traffic. Cost on a per-GB-out model"

I have no idea what they charge pr gb for this cloud product however.

[satanspastaroll]

When you do outbound networking, you're sending it out to the internet, not mangling it within AWS's network

[pm90]

I get your point, but then why does AWS charge for inter-az traffic? That seems like an "egress but not really" kinda thing. If AWS/GCP stopped charging for this, customers would be incentivized to build HA systems and distribute their workloads across AZ's, which are a win for both customers and you (since capacity is now spread instead of stuck in a zone).

[ben509]

The doctrine for HA is that each AZ should be fully independent, and if you do that, your inter-AZ traffic is relatively minimal.

And I think the charges for inter-AZ transfer are to incentivize customers to do that.

Of course, to make them fully independent, you have to replicate everything, so you wind up buying several redundant copies of your system...

[pm90]

> Of course, to make them fully independent, you have to replicate everything, so you wind up buying several redundant copies of your system...

Yeah, and keeping around warm systems ready to failover in case of a zonal outage seems like a preposterous waste of resources.

The alternative... to keep around multiple replicas of your system in different zones, all ready to accept traffic and which do serve traffic, seems more practical and less wasteful.

[erik_seaberg]

This. Paying for n+2 capacity is really expensive when n=1, but pretty reasonable at n=5 or n=10. Until someone gouges you on data transfer …

[satanspastaroll]

It's not a waste when it serves a purpose. Availability is a big concern

[pm90]

Sure, from the perspective of availability, it makes sense to keep this around. The US military has redundancies in place to handle many kinds of adverse scenarios, which comes at a price, but is justifiable. The point I'm trying to make is that if availability is the _only_ value, it becomes hard to justify that if you're a scrappy for-profit corp looking at your bottom line.

If instead of availability being the only value, there would be a more value provided from actually using such resources, more folks would adopt cross-AZ architectures which would be a win-win for both the customer (get HA for lower or no cost and go down less often and succeed in the market) and thus the cloud provider (keep raking in the steady cloud revenue as the customer grows).

[zmmmmm]

> so you wind up buying several redundant copies of your system

This is one of those gotcha's that company's hit. They see the public pricing page and think "wow that is much cheaper than one my internal IT department charges for X", and then when they go to actually implement they find that "best practice" says they basically have to more than double or even triple the cost to get a reliable system (more because not only do you have to duplicate all the infrastructure into a second AZ, you are getting charged for the replication traffic between them).

[jann]

But in all fairness, if you actually implement that ”best practice” HA infrastructure, you will also be miles ahead of almost all internal IT departments.

[late2part]

Inter-AZ traffic is Metro Area Network ("MAN") traffic. There is a cost for running network between locations.

This explains the cost.

Price probably should be based on value, not on cost. Why do they charge for it? Because they decided it's a good way to make money and profit.

[geggam]

Something has to pay for the fat pipe between colos ( AZ's)

[dx034]

But those are probably far below $1/TB. Otherwise other colocation providers couldn't offer that even with peering.

[nphase]

The bandwidth alliance has a response to this approach: https://www.theregister.co.uk/2018/09/26/cloudflare_bandwidt...

[satanspastaroll]

Outbound bandwidth also happens to be an excellent place to put any markup you can, as that also locks people from migrating away so easily

[CommieBobDole]

Because AWS is designed to be like a pitcher plant, shaped like a funnel with inward-facing hairs to make it easy to get in, difficult to get out.

That's pretty much it - in my experience most of AWS' awesome features are designed to lock customers into an environment where Amazon increasingly provides all of the components and services you need to do business.

I would assume the endgame is to create an ecosystem where the vast majority of customers allow their IT function (infrastructure, developing software, engaging third-party SaaS vendors, etc) to atrophy entirely, after which they'll have no choice but to buy what Amazon is selling, at any price, in perpetuity.

[jrockway]

It has no basis in reality. You can buy a 10Gbps transit circuit for $1500/month. That's $0.00046296296 per gigabyte.

[danpalmer]

That depends on how much saturation you can get. Most end users/companies can’t saturate this reliably, or have much spoiler traffic. Only the biggest players can minimise their seasonality enough to get close to this, and doing that is a service that it’s probably worth paying for.

[mlyle]

Who cares if you can saturate it? Breakeven is 18 terabytes/month, or 0.6% utilization on that line.

You could do this even just pegging the line 9 minutes per day and otherwise leaving it unused. ;)

[EB66]

+1 for this.

People forget just how affordable it can be to maintain your own infrastructure. You can have the hardware and network capable of supporting 10X your average traffic loads and still have it operate far more cost effectively than the equivalent traffic on AWS.

At my work, we slashed our overall hosting costs by moving a data warehouse off of AWS and on to our own self-maintained infrastructure.

But with a bloated inefficient IT department or non-savvy negotiations with hardware vendors or transit providers, it can also be more expensive than AWS.

[geggam]

I have had this debate with so many people, AWS is not cheaper than your own hosting. Period.

What you get from AWS is the logistics pipeline is already built as is the infrastructure should you suddenly require to serve factors of traffic more.

The ROI of AWS comes from the backend and capex vs opex debates.

Example : The CFO can go to the board and explain we are getting ready to reduce opex by laying off 10 developers at 150k yr during any meeting. The capex cost is usually fixed and hard to explain away.

[satanspastaroll]

I think the best cases for AWS is counter to marketing, very small companies. Their native offerings do make things great for a while when you're starting out.

The other is stupendous burst activity, like you just need a thousand(s) cores for a couple hours. Of course this doesen't mean the baseload has to be in AWS, just easy for small teams.

[K0SM0S]

I hear more and more cases of businesses moving parts off of the cloud, back on-premises. I think, in a wider perspective, it's going to play back and forth. The 2010's were 'forth' towards the cloud, the 2020's might see a peak back on-prems, etc.

It's generally a function of network cost versus infra cost, and so the 'equation' solves differently based on the longest tech cycles, from inception to maturity to skill pool to diminishing returns and then back again on some other mode — this really is a decade+ thing.

Some "always true" inherent advantages of one approach (e.g. availability for cloud, or resources for on-prems) would remain across cycles as permanent gains; disruption then occurs when some new approach (e.g. containerization) fundamentally upsets the order of costs.

[snark42]

> It's generally a function of network cost versus infra cost, and so the 'equation' solves differently based on the longest tech cycles,

It used to be OpEx was easy, CapEx was hard to get approved. As people took advantage and OpEx went through the roof people are getting alot more pushback on reducing OpEx.

[paulddraper]

> Only the biggest players can minimise their seasonality enough to get close to this

Okay, like AWS

> and doing that is a service that it’s probably worth paying for.

Okay, but how many orders of magnitude?

[bsder]

Well, reverse it: Amazon is about $1K per 10TB.

That's an expensive disk drive.

[nostrebored]

Thinking of it as a disk drive is fallacious

[nine_k]

Because this is what the market agrees to bear.

They want you to keep all of your data in their cloud, do all your processing there using their services (because doing it elsewhere incurs expensive egress fees), and get paid handsomely for your need to actually serve the data to your customers.

This also makes a migration additionally expensive, because you would need to egress all your data (old logs, some fresh backups, etc) in a short while.

So it's basically a soft lock-in.

[nickjj]

> Because this is what the market agrees to bear.

I think it's more than that because a lot of folks aren't aware of the costs until they need to or want to move and then they get hit with a massive bill.

So maybe they chalk it up to experience and pay the bill because they have no other choice or calculated it's still better in the long term.

To me that's a much difference scenario than walking into a store and happily buying a dozen eggs for anywhere between $1 and $1.50. In this case you know what you're getting into before you make the purchase and everyone around you (other customers and businesses) decided that's what eggs will sell for in the open market. With outgoing data fees, it's more like a "take it or leave it" price dictated by the provider while they already have your data and there's no price competition since they are the sole business with your data.

Whether or not it's the customer's fault for not doing enough research is debatable, but it certainly doesn't help that most providers make it pretty difficult to calculate costs.

[kick]

That's still the market.

The market isn't exclusively good, and has many failure modes. This is one of them.

[nothrabannosir]

While true, this dilutes the value of the statement. At that rate every price point “is the market”, and nothing of further interest can ever be said about counter intuitive pricing.

Sure, it’s the market. Why? Is it a Giffen good? Is it a case of very poor visibility of services to consumers? ...? There is something interesting going on, let’s figure it out.

[satanspastaroll]

There are some bandaid-solutions people have used to migrate away from this without taking a major hit. One is utilizing the free outbound bandwidth from lightsail, and bigger transfers can use snowball which is 'only' $0,03/GB outbound + shipping

[kuschku]

A major issue is that in most markets outside of Europe, peering is less common, and transit prices are significantly higher.

Europe is probably the most competitive and best market in this case, almost all other markets are dominated by monopolies. Yes, even significantly worse than the Telekom monopoly.

Look at transit in Singapore, Japan or Australia. Or even in Brasil. You’ll go bankrupt even from trying to deliver a single movie to customers.

[iptrans]

Actually Singapore and Australia aren't that terrible anymore.

Hurricane Electric has POPs in both Singapore and Australia. Transport between Singapore and Australia is $1.50 or less. Peering ports are $0.20 per Mbps.

If all you want to do is push movies at customers, there are plenty of dedicated server providers who will sell you bandwidth on the cheap in both countries.

Obviously YMMV if you want better routes or direct interconnects with local monopolies.

[kuschku]

Well that's exactly the issue, HE often doesn't have peering with local monopolies, or at best only a 10G link, because those monopolies don't even care.

[qaq]

Thats one of the big money makers plus it locks you in so there is 0 chance they will change it.

[berns]

> Is there a plausible explanation why egress fees from cloud providers costs around $0.1/GB?

Isn't DigitalOcean a cloud provider ($0.01/GB)?. Isn't Oracle a cloud provider (first 10 TB free, $ 0.0085 after)? Isn't OVH a cloud provider (free bandwidth)?

[dx034]

Which means higher pricing at GCP/AWS/Azure can't be explained by higher availability, DO offers the same.

[semiotagonal]

Unless I'm making a mistake someplace, Digital Ocean includes 1TB egress to the internet from a $5-per-month host. On Google Cloud Platform that egress would cost $120/month by itself.

https://www.digitalocean.com/pricing/#Compute

https://cloud.google.com/compute/network-pricing

[privateSFacct]

What you find is if you spin up 1000 of these $5 hosts (ie, $5000 per month which is pretty) and try to egress 1PB per month you can't. Either they disconnect you, the systems can't actually handle it or some other gimmick.

One thing folks like about AWS - you can actually know what you will be paying and there is no fake / hidden limits.

That's my somewhat outdated experience wasting a TON of time on this idea ages ago.

[Dylan16807]

But going back to the original number of $0.01/GB, that's what they state upfront as a bandwidth overage cost. I would be very surprised if they disliked customers paying the overage fee. And that's still only $10k per petabyte, vastly cheaper than AWS.

Give them a warning if you're going to spike your bill that high, but there shouldn't be any fake/hidden limits.

[ikiris]

When your link is bargain basement C* "good luck" peering, it can be super cheap and honestly will likely get everything most users need.

When you care about packet level SLOs... yeah, you start to shop around.

[dx034]

But how many applications on AWS really need that? It's probably a fraction of bandwidth that'd need such a SLO.

[throwaway5752]

I was just reading about advisor termination fee terms which were some substantial multiple of all fees incurred over some past number of years.

People under pressure make short term optimizations at the expense of long term strategy. There's nothing more to it. AWS reduced compute and IT expenses in the next few fiscal years, so people jumped all in on it. Solve the problem now. Someone will fix it in the future.

[asdfman123]

Programmers should campaign for hefty termination fees. Sure, you can fire me, but you'd have to pay me one year's salary.

Actually, it would probably just make some programmers even more intolerable.

[wolco]

Instead of options ask for severance term next job interview

[tallanvor]

I doubt that you'll find you can consistently push or pull 1Gbps from a Hetzner server. I certainly can't. --That's not to say that they're a bad deal, but their total bandwidth is nowhere near what the major cloud providers can give you.

[nh2]

Cannot confirm this claim.

I've always got full 1Gbps out of Hetzner, even across the ocean, and for periods of multiple days of transfers.

(Like on all machines, you need to set the right TCP settings for windows sizes to make it physically possible.)

[dx034]

Why can't you? I'm doing this regularly (backups & restore). I get reliably 1gbit/s from other providers, also from US (with multiple connections of course). Haven't come across a provider where speed would be significantly below that.

[tschellenbach]

Also Hetzner offers hosting that works sometimes/ maybe. I only consider it a viable option in the earliest days of starting a project.

[js4ever]

I have few dedicated and few vps there, this year I had less than 5 minutes of downtime on 1 of the vps and 100 % uptime for all the rest, do you have more details about your experience?