|
|
|
|
|
|
by jeffbee
892 days ago
|
|
|
> With Ethernet, applications and network stacks (the usual kind — see below) ought to do their best to control congestion Exactly. Network endpoints infer things about how to behave optimally. Then they put their frame on the wire and hope. The things that make it possible to use those networks at high load ratios are in the smart endpoints: pacing, entropy, flow rate control, etc. It has nothing at all to do with the network itself. The network is not gold plated, it's very basic. |
|
|
A ratio has a numerator and a denominator. I can run some fancy software and get (data rate / nominal network bandwidth) to be some value. But the denominator is a function of the network. I can get a few tens of 10Gbps links all connected together with a non-gold-plated nonblocking switch that’s fairly inexpensive (and even cheaper in the secondary market!), and each node can get, say, 50% of 10Gbps out as long as no node gets more than, say, 50% of 10Gbps in. That’s the load ratio.
Or I can build a non-gold-plated network where each rack is like this and each rack has a 100Gbps uplink to a rather more expensive big switch for the whole pile of racks, and it works until I run out of ports on that big switch, as long as each rack doesn’t try to exceed the load ratio times 100Gbps in aggregate. Maybe this is okay for some use case, but maybe not. Netflix would certainly not be happy with this for their content streaming nodes.
But this all starts to break down a bit with really large networks, because no one makes switches with enough ports. So you can build a gold plated network that actually gets each node its full 10Gbps, or you can choose not to. Regardless, this isn’t cheap at AWS scale. (But it may well be cheap at AWS scale, amortized per node.)
And my point is that egress does not add meaningful cost here. A 10Gbps egress link costs exactly as much, in internal network terms, as a compute or object storage node with a 10Gbps link. For an egress node, it costs whatever 10Gbps of egress transit or peering costs plus the amortized internal network cost, and a compute node costs whatever the cost of the node, space, power, cooling, maintenance etc costs, plus that internal network link.
So I think there is no legitimate justification for charging drastically more for egress than for internal bandwidth, especially if the cost is blamed on the network cost within the cloud. And the costs of actual egress in a non-hyperscaler facility aren’t particularly secret, and they are vastly less than what the major clouds charge.