[cperciva]

I think this is one reason why EC2 is limited to 25G while 100G has been commodity for a long time.

Interestingly, the ENA driver has #defines for speeds up to 400 Gbps.

My guess as to why EC2 instances are limited to 25 Gbps is that it's a matter of balancing overprovisioning and the need to avoid having a single instance eat too much of a rack's bandwidth. I don't know how much bandwidth they have going to each rack, but there's a limit to how much it makes sense to provision; if typical bandwidth is on the order of 10 Gbps per rack (say, 80 instances pushing 125 Mbps on average) then you might want to provision 200 Gbps/rack and limit each instance to 25 Gbps rather than provisioning 1 Tbps/rack and limiting each instance to 100 Gbps.

(Numbers above are completely invented; I don't have any internal knowledge of how Amazon's networks or datacenters are set up.)

[kev009]

Most large operator datacenters are converging toward things like Clos and fat tree networks that provide abundant bandwidth at acceptable cost and with minimal blocking. Switch silicon vendors have really done yeoman's work pushing the envelope to make this possible and inexpensive. AWS might have such magnitude of machine count and generally low customer resource utilization that they can oversubscribe a lot, but it would be pretty silly to only bring in 200gbps to a rack post 2014 when the Broadcom Tomahawk switch ASIC became dominant.