[kaliszad]

You cannot use IPv4 100% efficiently. Even if you approach 90% usage, that would probably be very optimistic. AWS has probably millions of physical servers by now (they had ca. 1,4 million servers by December 2014 it seems) each with probably hundreds of services each needing an IP. Yes, you can stretch 10.0.0.0/8 with NATs all over the place but at some point, you need the public IP. With that kind of appetite you need multiple /8s of IPv4 space just to keep things running. I don't think they are crazy buying that much address space and concurrently improving IPv6 support - they most likely really need it.

[tudelo]

> You cannot use IPv4 100% efficiently.

Can you expand on this a bit? I'm not a networking guru but it's my understanding that an IP address is an IP address, how could it be used more or less efficient?

[ArchOversight]

Each time you subnet you lose three or more IP's.

A /24 has:

- .0 - network address - can't be used

- .255 - broadcast

On top of that you generally need at least 1 more IP address that is the gateway for that network.

- .1 - Usually

If you have a network with fail-over gateways generally you need to assign them individual IP's, so you end up with:

- .1 - floating IP

- .2 - router 1

- .3 - router 2

If you end up subnetting down into small subnets to give customers only let's say 16 IP's, (so a /28 (32 bits - 4 bits)) customers can only use 13 of those addresses (network/broadcast/gateway are already taken).

This gets worse as you go smaller, because each time you subnet you end up losing more IP's to the network/broadcast/gateway.

[rachelbythebay]

Consider what happens if your machine has another IP address somehow (a so-called "primary"), and then you add all of the "additional IPs" as /32s. Bingo, you can now use all of them.

The rest of the network doesn't care what they look like at certain scopes. Those devices just know to route it to the next hop, and when that next-hop is your box with a bunch of interfaces/aliases/whatever configured, it'll just handle it.

Try it, you'll like it.

(Edit: IPv4 here, hence /32. No such foolery needed with v6.)

[ArchOversight]

Yes, absolutely. Unfortunately there's still some wastage as you end up having to set up a public IP for both endpoints to allow the routing to be public. (Yes, you can use private IP's for routing, but it makes diagnosing issues much more difficult). So then you end up with /30's used for point to point links, which is 4 addresses.

You can use /31's for point to point if both devices support it, but it's still hit or miss whether they do or not.

Even when getting a block of IP's from providers to a customer edge (think ISP's like Comcast or others) they tend to require using the .1 as the gateway, and the others are considered on-link and thus there is a network/broadcast address.

Your suggestion doesn't allow direct L2 communication between machines without sending traffic to the router, even if the two systems are L2 adjacent.

Even with v6 you'd want to assign multiple IP's to the same interface, except that instead of getting /128's routed to a machine you just a get a single /64 and you can use any of the addresses out of that range.

[luhn]

All public traffic is NAT’d through AWS’s Blackfoot servers, EC2 instances are not directly assigned public IP address. (you won’t see the public IP in ifconfig, only the private IP) So there is little to no subnetting happening in Amazon’s public IP space, as a small cluster of machines are assigned the entirety of the DC’s public IP block.

[ArchOversight]

I don't disagree with that, I was simply explaining to the asker why IPv4 can be wasteful when it comes to how IP space is allocated usually.

[AmericanChopper]

I’ve been assigned a .0 public IPv4 address by AWS before. I was a bit confused when I first saw it, but it worked just fine.

[ArchOversight]

Sure, there is no problem with using a .0. For example in 10.0.0.0/23 you'd have

10.0.1.0/32

As a perfectly valid IP address.

Also, routing an entire block allows you to use all of the IP's in said block. So there are ways to do it efficiently, but with routing a block that block is not considered on-network and thus doesn't have a broadcast address nor can it use one.

[adamcharnock]

I had this too. I assumed it would be fine - and it mostly was. However, we tracked a weird bug with some of the companies IoT products not being able to open a connection to the .0. Never figured out if this was a dodgy IP stack on the devices, or if it was the particular mobile carrier in that area of the world. My money would be on the latter though.

[pixl97]

Yep, you can run into weird issues at times with devices/firewalls that are misconfigured. Saw more than 1 admin that would drop anything from a .0 as "it is a bad address and nothing from that address should cross a router".

[gene91]

That works when the subnet is /23 or larger.

[outworlder]

> If you end up subnetting down into small subnets to give customers

Since this is about AWS... they don't do that. They will assign a random IP out of their available public IP for every resource that needs an IP. For many of these you can allocate an IP and move it around if you want (that's why they call it 'Elastic IPs').

Unless you have special needs (and actually own the IP ranges) your VPCs will all be in a private IP range. You can even have multiple VPCs with the same range with zero issues (unless you want to peer them).

[unilynx]

To keep things manageable and prevent any broadcasting node from reaching every system you have, you generally slice your address space into smaller networks

Each network needs at least one gateway inside that network to reach the rest of the world, so that IP address is lost. Conventionally you also lose the lowest and highest IP address in a block to the network number and broadcast address (you can free those up, but it might confuse devices or admins)

So for every split or subdivision you make, you lose at least 3 addresses.

Conventionally you also split a network at a specific bit boundary (CIDR) so each network has 2^n IP addresses in it. If you went for n=4, you have 16 addresses and already lost 3 to the above factors, so now you have 13 addresses left.

If you happened to have 14 machines attached to that network, you need a network with 2^5 = 32 IP addresses, and you've got 15 unused IP addresses with no way to give them back to a higher level.

All quite similar to why a phone area code may not be able to make use of all phone numbers inside that area, but have no way real practical way to give them to other networks.

[CountSessine]

Routing tables get bigger and bigger as the topology of the addresses stops being a good generalization of the topology of the network. As you slice and dice networks into smaller different globally assigned networks, routing tables need new entries for the smaller networks.

Pretty sure that we’ll never see routing tables with 4 billion entries. So it has to end somewhere.

[samoa42]

https://tools.ietf.org/html/rfc3194

[outworlder]

> AWS has probably millions of physical servers by now (they had ca. 1,4 million servers by December 2014 it seems) each with probably hundreds of services each needing an IP.

Why would any significant fraction of these servers require public IPs? And services for that matter.

I wouldn't be surprised if most of them are in private VPCs with only very few endpoints exposed.

If you are referring specifically to AWS, having servers in public subnets is actually an anti-pattern. You may want to do so with bastion hosts and a handful of more specialized services. For everything else, put them behind a load balancer. A single NLB will take one IP per availability zone and will be able to service hundreds of servers, if not more.

Then you have things like Cloudfront and the like.

Not many IPs are needed, overall, compared to the number of actual servers.