Hacker News new | ask | show | jobs
by fulafel 998 days ago
It seems obviously against AWS incentives to offer working v6 - all their influencing tools ("well architected" criteria, certificates) strongly herd you towards building mazes of ambigously addressed 10.x RFC1918 networks, and not internet style architectures with end-to-end addressing.

In the world of their recommendations, even the concept of a "public ip address" is a red flag, and AWS even recommends (for an added cost of course) tooling to flag and "mitigate" them. These provide a strong lock-in effect when customers spend effort to build the complex infrastructure for them in the name of security, even though in reality they hurt security through unnecessary complexity, addressing ambiguity, etc.

8 comments

Azure is copying this wholesale.

I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale.

Literally none of those products would be required if they had just made IPv6 work properly.

Instead, they NAT IPv6, so you can't even use it to avoid the NAT forced upon you by IPv4. They also release new products -- in 2023 -- that don't support IPv6 and likely never will.

There is still an entire page[1] of listed limitations in the docs: https://learn.microsoft.com/en-us/azure/virtual-network/ip-s...

Think about how insane it would be if this was the IPX -> IPv4 transition. Imagine if this page said "IPv4 limitations: All VMs must include at least one IPX address, etc..."

Sounds nuts, right? I was migrating customers to IPv4 from IPX in 1999, and IPv6 support materialised in about the 2001-2003 timeframe. It's been decades, but it still feels like 1999 and migrating Novell NetWare where we had to have IPX+IPv4 because "not everything supports IPv4 yet".

[1] This is definitely not all of the limitations. Most of their PaaS products don't support IPv6. Hence, any IaaS+PaaS solutions must use (mostly) IPv4.

A lot of IT folks are still fearful of IPv6. I've been on calls where people disable IPv6 as a matter of "best practice." It's sad. People will gladly learn the latest flavor of the month web framework but won't take time to gain experience with a fundamental protocol.
I probably fall into this bucket but I’ve seen disabling it fix very odd issue entirely too many times to write this off as just “sad”.

Learning a new framework doesn’t (often) break things for the end user in a way I can’t diagnose/reproduce. IPv6? Totally different ball of wax.

Also my ISP, a fiber gigabit provider and the best available in my region, doesn’t support IPv6. Until they, and others like them, get on board fully I don’t see customers having a snowball’s chance in hell of working cleaning with IPv6. I can only imagine how long it will take the shitty ISPs to fully support it.

Why? What's the (supposed) fear?
Why? What's the (supposed) fear?

Disclaimer: I have never worked for Amazon but I can add some input from the assorted medium to large companies I have been employed with. I won't mention any names. This is for someone I know reads my comments wink wink

I am not justifying it, just adding some of the bits I experienced. There are many security devices that do not have the same capabilities on IPv6 as IPv4 yet. Some enterprise IoT devices only support IPv4. Adding to this some network engineers don't want to step outside of their comfort zone and tooling/scripts to generate configurations automagically do not yet support IPv6. As the company grows they hire less Sr. Network Engineers and some of the new people depend on but do not understand the automation. e.g. someone wrote some API and they retired or changed companies. Some tooling may remain stagnant for some time. It's also a heavy lift to retrofit some enterprise environments for smaller changes so people fear the outages they will induce implementing IPv6. In some companies it is a major change just to migrate customers to a new load balancer endpoint. There may also be hundreds of undocumented things due to employee churn and lack of change control running that when broken will cause extended outages. And then there is internal politics and finger pointing...

Again, not justifying it, rather I think there are too many moving bits and complexity that people have added over the decades and they are paralyzed by fear and risking the loss of their paycheck. And then there is the embarrassment that comes from having to acknowledge that nobody knows the current state of an environment and that embarrassment can go all the way up the organizational chain.

That is based on my experience of being brought into companies with the speicifc task of, "Hey, make this simpler, reduce outages." It's rarely strictly a technical challenge but rather having to navigate politics, personality types and individual sub-org leaders that have had independent control of their environment for a long time. The more I think about it this could be a topic in and of itself how companies induce self inflicted bloat as they grow.

The biggest issue is IPv6 is a privacy, wide open wild west, there is no privacy on IPv6. Every device's IP is literally public, on the public Internet, 24/7. All so called privacy extensions or improvements do not change the lack of privacy of IPv6 and one more thing, the address structure sucks.
This is wrong. Every IPv6 interface has a link-local address (which is not routable outside the LAN) as well as a global address. Global addresses are just addresses that come from a block allocated by the IANA. It has no bearing on whether the interface is reachable from the public internet. Just like with IPv4 networks, a stateful firewall will prevent unsolicited inbound connections.
FWIW this does not have to be true for companies that do not wish to expose internal nodes. I'm not even talking about the privacy extensions. I realize that people beat the drums that one must not NAT IPv6 but it can absolutely be a NAT just like IPv4. I would actually expect in most companies that they don't even add IPv6 inside their datacenters, rather they just put a block of IPv6 addresses on some load balancers at the edge and then point their edge devices L4 and L7 mapping to IPv4 nodes in their datacenter. The same could apply to VPN/WAN configurations in some cases much like how mobile networks are configured nowadays. There would need to be some of this regardless due to ACL's required between companies that don't use network VPN's for restricted end-points. e.g. PCI to PCI environments.

In the early days of IPv4 many big companies did not NAT IPv4. I was at a company that did this. Our workstations all had routable public IPv4 addresses. There are still some companies that do this. One of these was a company that had 20 managers and VP's on a call when I just wanted to give their network engineer a CIDR block and a pre-shared secret for a network VPN. I suspect they will be using public IPv4 addresses internally forever. And I doubt they will ever sell their /8's unless people comment on their Vogon poetry.

Amazon is probably one of the exceptions as they have so many geographically disperse configurations it would be harder to continue using RFC1918 address space. Not impossible, just difficult. I can think of a few other big companies that do manufacturing that are spread out around the world that would probably run into walls with RFC1918 at some point. I've seen some of them take over public address space for internal routing which then breaks access to some things on the internet thus requiring double/triple NATs. 1/8 assorted, 25/8 MoD, 26/8 DISA are a few I've seen.

IP-based "privacy" is an illusion. With IPv4, your public IP (NAT router IP address) may not change for months, years, and possibly not until your change your router/MAC address. With IPv6 privacy extensions, your address changes regularly. This seems like an improvement.
It's definitely "best practice" to turn off features you don't need. Who knows, maybe in five years someone will find a bad bug in the implementation of the IPv6 stack, then you'll be glad you decreased your attack surface.

Not saying this is what you should do, just a common rationalization.

It's also "best practice" to learn a new, foundational technology (like IPv6) sooner than later, perhaps less than 20 years after it was first available.
I know IPv6, just don't feel the need to use it. I prefer to have one firewall and public network interface to worry about. I can't disable IPv4 yet, so the logical solution is to disable IPv6.
Right and what is better way to learn than deploying random features to prod!
It's simple fear of the unknown. Many folks haven't learned it, so they just turn it off... Ignore it, kick the can down the road.
It's individuals responding to the incentives before them. If something goes wrong because they're using IPv6, it's their fault. If you never upgrade anything until you're forced to, then you can never break anything by upgrading, and you're never seen as a "stuff breaker".
This is it. Having broken stuff by upgrading myself, I can say it does not make you popular. This is especially the case if what the upgrade did was tighten up some bug that it turns hid a bug in your team's code...
I have a reason: we do per IP rate limiting. It's easy enough for IPv4 when the number of IPs is necessarily not too big to fit in a small redis for example, but for IPv6 everyone have at least a /64.

I'm curious how people do it btw, if you have tips to share, I'm all hear. Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.

Given that the only routable IPv6 address space is in the 2002::/16 range (is 2003:: in use yet?), and the standing recommendation for ISP CPE endpoints is to allocate a /48 per customer (a customer can't do any local subnetting if only allocated a /64), the effective address space for rate-limiting is the exact same size as the current IPv4 address space: you only need to track bits 16-47.

It's possible that cloud providers assign smaller ranges to their customers, so you may need to allocate more bits for granularity in that case; on the other hand, one might naively assume that cloud providers are more responsive to abuse reports than ISP's.

While the recommendation is to hand out a /48 to each individual customer it's definitely not the standard.

Cox & Spectrum only hand out /56. I'd hate to be banned because my neighbor did something bad and we happen to be in the same /48.

When a bunch of households or cell phones are on the same IPv4 do you have any measures to compensate?

> Do you simply rate limit IP ranges? Even limiting per /64, it's still potentially quite a lot of /64 to track.

Yes you'd limit by /64 or slightly larger.

The live set of IPs shouldn't be very big.

We put limits high enough that it's far enough for any expected usage, including a bunch of users on a single IP. If we see rate limiting happening in practice and it doesn't seem to be an attack, we revisit.
Answering myself: I found this interesting article https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/
You treat ipv6 /64 just like /32 in ipv4
Mainly that everything is publicly routable.

People see a 10.x and instantly know it can't be reached from the public internet. IPv6 is much harder. For internal-only stuff there is the fd00::/8 block, which AWS actually does use, but there is no equivalent range for outgoing-only connections.

Because there's a lot of shit that still doesn't work well with IPv4. Logging is one good example - a lot of software that uses its database for event logs has the database column for remote_ip defined as VARCHAR(15), you can guess the rest of what happens when deploying that with IPv6 enabled.
These things take time. I mean, it's only been 25+ years since the first IPv6 RFC was released...
It’s a big scary number and the letters just break understanding.

There are some annoying operational issues around it as well, common one: DNS hostnames for devices that only do SLAAC

You wouldn't want to do SLAAC and hostnames.

If your network card breaks you switch it out, and you make sure your IPv4 settings apply to the new card.

If you fully rely on IPv6 you'll get a new address.

And if your devices self-update DNS then you have to make sure they pick the right address, as there can be many due to privacy extensions.

Lastly, combining privacy extensions plus hosting stuff is hard, as you don't know on which address a certain port is bound.

In the case of my corporate VPN, Teams doesn’t work with IPv6 enabled. Something on either the MS side, or the SSO one.
I was troubleshooting some bad routing in a service. Some endpoints worked fine most of the time but would occasionally lose packets for a bit.

Turns out it's a bug in the kube stack when containers have both ipv4 and ipv6 addresses.

This was last year.

I suspect the ipv6 cases aren't being tested very well across the software stack.

Some current Apple products still have problems with IPv6, and the standard practice is to disable it entirely.

I continue to do that on all my personal equipment.

Comcast at some point stopped letting you administrate your own router. You can log in to it, but port forwarding is no longer available through the administration interface.

If you want port forwarding, they recommend that you do... something. It's not clear what; what you can find on the internet is mostly just people complaining that they insisted to customer support that they needed port forwarding, customer support said they'd do that, and port forwarding still doesn't work.

But, intriguingly, it turns out that a Comcast router will also assign every device on your network at least one public ipv6 address. They also firewall all incoming ipv6 traffic, but unlike the situation with port forwarding, you can disable that firewall on the router admin page. (You can't put up your own firewall.)

> port forwarding is no longer available through the administration interface.

Sounds like the CGNAT experience, possibly blanket applied even if you have a globally routable public IPv4 in order to have consistent behaviour and reduce network management / support case complexity.

I’m also a CrapCast subscriber.

I bought my own (plain) modem and manage WiFi/routing myself.

I get the direct Internet-facing IPv4 and IPv6 addresses.

No monthly equipment fee!

One cannot unplug the Comcast issued router, power cycle the modem, and plug in a customer owned router?
All I see are combo units these days.
You have to bring your own modem. Then you bring your own combo unit or bring your own modem that you can plug a router into.
All the modems I see support bridge mode. When enabled the Comcast device doesn’t do any of the routing at all. Your own device gets to do that instead.
Yeah, even at my father's place (I've had different providers than Comcast for personal reasons, but FWIW same story) I've never had a problem just plugging in a router of my choice and using that instead. Makes it a lot easier to handle quirky setups between modem swaps anyway.
It's almost as if those big cloud providers want to prevent a transition to IPv6 as relying on IPv4 will solidify their oligopoly.

Any new competitor would have to cough up a huge upfront cost for IPv4 addresses, if they could even get them at that scale.

Wait a second. A private link means that a service endpoint is public so a part of your traffic goes through the Internet, which is supposed to be insecure (even if you have encryption in transit?), so you don't want to do that and they will happily route your traffic internally so it is not exposed to the bad Internet - for a fee. All these VPC Endpoints etc. cost money and you are charged by the our. I don't think IPv6 would change anything here, they would find a way to charge customers for "security" anyway.
That's what I thought at first, but that's not quite right.

Service Endpoint: Allows a PaaS service (that itself uses public addresses) have firewall rules for overlapping private vnet addresses. E.g.: You can have have two VMs both on 10.0.0.123 addresses (in separate VNets) using individual "Allow" rules to the target service. Essentially Azure tags the traffic at the VXLAN level with the source network ID on top of the IP address, making it a "fat IP address" that is unique within Azure and can be used in firewall rules.

Private Endpoint: Makes a PaaS service appear on a private network address range instead of the default public range. This allows your on-prem firewalls to isolate your specific PaaS instance from other customers -- otherwise the traffic gets "blended in" with everyone else in the same public service tag ranges. This also allows you to use your ExpressRoute fibre links to route traffic from on-prem to the public service.

In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. You have to go out of your way to route traffic "via the Internet". Remember: Network addresses are just numbers! Routing rules determine how they flow, and public addresses can be used on private networks.

Fundamentally, all this exists just to enable the ability to firewall things. With overlapping IPv4 addresses and small shared blocks of IPv4 addresses with NAT behind them, it would be impossible otherwise.

With IPv6, using firewalls would be much simpler because overlapping addresses aren't needed any more. Similarly, PaaS services could trivially allocate IPv6 addresses per customer instance, so that customers could apply selective firewall rules.

> In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone. You have to go out of your way to route traffic "via the Internet". Remember: Network addresses are just numbers! Routing rules determine how they flow, and public addresses can be used on private networks.

My understanding is that if you don't have a private endpoint, your traffic to an Azure cloud service won't be routed out to the "big bad internet" per-say, but it will be routed within the Azure AS as mere IP traffic.

If you have a private endpoint to an Azure service in your virtual network, that means Azure has provisioned you a virtual NIC with some private IP address, and presumably alters DNS resolution within your network for that Azure service to resolve to the IP address of the NIC. The NIC provides (presumably encrypted) link layer transport out to the Azure service.

Compliance for some customers may dictate that there aren't any routes out to the public IP address space from within a network. If you still need access to cloud services, private endpoints are a necessity.

All that to say, I think Private Endpoints provide more than just a means of firewalling traffic/changing the IP address associated with a service; the actual transport from client->cloud service is fundamentally different.

You’re exactly right that the main point of private endpoints is to allow customers who aren’t allowed to open firewalls to public internet to still connect to their public services like Azure Storage, Key Vault, or SQL
> In all scenarios, the traffic goes over Azure networks and/or Microsoft's private backbone.

That's the whole selling point of private links. How could you possibly missed that? That's exactly why companies onboard onto the service. They say exactly this exactly on the marketing brochure. That's why customers line up to pay for it: to get their traffic flow only through private networks instead of through the wild.

What kind of confusion reigns in your mind to come to the conclusion this was some obscure conspiratorial gotcha?

It boggles the mind how you felt the need to come up with absurd conspiracies involving IPv4 to arrive at a claim that the marketing pamphlets show front and center as their whole reason of existence: avoid traffic to go through the internet, and instead pay extra to go through private pipes they own.

It's as if the name actually means something!

> That's why customers line up to pay for it: to get their traffic flow only through private networks instead of through the wild.

I believe their point is that it doesn't flow through "the wild" in any case, it is probably routing within the Azure AS(aka Microsoft controlled networks). However, as you say, people line up to pay for it, likely for reasons having to do with their architecture/security model/compliance requirements.

What a shit show. Seriously I can never rant enough about how awful Azure networking and their bullshit concepts is. Like they don’t know how to do networking, so they’re gonna introduce a bunch of shit and pretend that nonesense makes perfect sense because of their own ineptitude.
Most of the complexity is a side-effect of having to do IPv4 far past the scale where the ~17 million private addresses might be sufficient.
Some of the complexity seems to be the security theater of the lowest common denominator of customer demands. Companies invest too much money into incredibly expensive Palo Alto firewalls then demand Azure route through those too so that their cloud operations are as theatrically "secure" as their main network because look at all those amazing sunk costs invested in it.
> I've lost track of all of the "Private Endpoints", "Private Links", "Service Endpoints", "Private Resolvers" and "Virtual WAN" products they've introduced... all to make IPv4 work at scale.

I'm not sure what you've been reading, but the concept of a private link has absolutely nothing to do with IPv4 vs IPv6. In fact, practically all your remarks don't involve the issue at all.

The most charitable interpretation of your comment is that you're making the mistake of conflating any application involving a virtual network as something caused or involing IPv4.

I replied to someone else's similar misconception of what problem Private Link solves here: https://news.ycombinator.com/edit?id=37609614

Hint: it's a complex workaround for insufficient IPv4 addresses.

I think your comment shows a high dose of ignorance and complete lack of research on the topic. The whole point of private link is to not route traffic over the internet, and instead flow traffic between private networks through private pipes.

One of the primary usecases and design requirements for this service is regulatory compliance. They say right on the tin that the service is designed to send traffic over private networks, including AWS's own global network. The whole point of private link is to ship data through the pipes you own, instead of routing it through the wild. I don't know how you could have missed that.

More importantly, you really need to want to use private link connections. This is a value-added service. You need to want to go out of your way to avoid your traffic to go through the internet to onboard both ends of your services to private link.

Not only are your conspiratorial hypothesis completely out of base, even your baseless assumptions have absolutely no relation with what version of the IP protocol is in place.

I'm the first to join in on any good old fashioned AWS/big cloud provider bashing, but these should be grounded on reality.

If you think that either Azure or AWS loop terabits of customer traffic between two of their own services "out to the Internet" and back just because the IPv4 octets don't start with a "10", then you're the one who's missing the big picture.
Yep. Pick your poison. You're either nickle-and-dimed for a "NAT gateway" or overpaying for "VPC endpoints". Often, it's both. I preferred the EC2-classic days, honestly.
I am at a total loss to understand the link between this post and the prior one - can you explain?
Private link is a clever way to implement real network segmentation

That is, when you have a customer in some network and a provider in another network, you had to implement full connectivity between the customer and the provider

With private link, you can remove all that connectivity, and instead expose the provider' service to the customer The service, nothing more, so just one endpoint

This is really good from a security point of view, but also for managing your stuff (especially if there are multiple teams in the compagny): because you now have a resource, you can easily list the services you expose to other people, and whom are your customers

The IPv6 version of this would be ...

1. get list of customer netblocks

2. setup "internal" service(s) for customers

3. setup firewall rules to allow customer <-> service allow list

4. setup DNS records

5. tell customers DNS and API targets

There is no "ipv6 version of this", private links have no business with layer 3 stuff
If you're not using token ring, it's not a real network.
I read that as "Tolkien ring" - which would also be appropriate, since "...and in the darkness bind them" is a good summary of cloud vendor lock-in.
"Nine /8's were gifted to the race of cloud providers, who above all else, desire power"
It's not just AWS. Microsoft, security auditors, penetration testers, cyber insurance companies, etc. also largely insist on not having publicly addressable endpoints.

I don't understand why, but until some large tech company starts pushing for end to end addressability as best practice, I have no choice but to follow the conventional wisdom to avoid throwing up red flags.

> Microsoft, security auditors, penetration testers, cyber insurance companies, etc. also largely insist on not having publicly addressable endpoints.

> I don't understand why […]

Excluding Microsoft, all the others find it easier to have as a checkbox to make it easier to confirm that "internal" hosts are actually (theoretically) internal since RFC 1918 isn't allowed outside.

Of course most companies' firewall and NAT rules are probably all sorts of complicated once you get to a certain size (never mind stale open-rules which were never cleaned up), so a bunch stuff is probably accidentally exposed. Also, most attacks are probably from compromised clients nowadays, so even internal hosts need to be locked down as the castle-and-moat security model isn't (as) valid.

But having "internal-only" hosts is low-hanging fruit on the security checklist.

> I don't understand why

I will resist the urge to be snarky at your expense and politely point out that exposing your LAN to public routing tables is madness, from all perspectives.

It brings no benefits and carries huge risks.

Is IPv6 Unique Local Addressing still a thing (or again)? Just because a machine has an IPv6 address does not mean it is automatically routable over the entire Internet.
>exposing your LAN to public routing tables is madness

And I don't understand why people think that.

You are exposing a /64 network. That's 2^64 addresses, no one can scan your LAN if that's what you fear, nor can anyone reach your hosts if you build a stateful firewall that denies incoming connections - you know, just like NAT. But minus the packet modifications.

> no one can scan your LAN

Are we really back to security by obscurity? Please don't tell me you are serious.

Anyways, you can't rely on ISP's handing out sufficiently large network ranges to make your security-by-obscurity scheme work.

Are we not _already_ attempting security by obscurity at the very moment we talk about "exposing your LAN" as a supposed weakness of IPv6?

/64 is the smallest network your ISP can hand out, of course you can rely on that. Even my mobile phone is getting a /64 from my ISP.

Using global addresses is not, of course, "exposing your LAN to public routing tables", or any charitable interpretation thereof. Reachability != addressing.
Global addressing is a bug and a ticking time bomb in this case, not a feature.
I work in Azure, but my experience is that customers want this - and for good reason. Customers want their own private network to prevent intrusions and exfiltrations, just on machines they don’t own. Or even better, put the nice fancy batteries included PaaS services in these networks too.
I see this at work, being within such a customer. People driving these mandates barely understand IPv4, let alone know what IPv6 is. They're software developers after all, not CCIEs.

There's nothing preventing you from having a private network using unique address space that's either blocked from accessing the internet via a firewall on a router or just plain not even routed. You could even use ULA networks with stateless prefix translation to avoid using GUA addressing for your private network.

The sad part is that IPv6 support is abysmal on every cloud so just migrating to it imposes serious limitations as addressed by the blog author.

> Customers want their own private network to prevent intrusions and exfiltrations, just on machines they don’t own.

If your (default) gateway from one network segment to another network segment only has one rule, default-deny, then it's not a problem. If you think that's not enough, then use IPv6 ULA (fd00::/8).

But why should the incompetence of some customers limit what all customers can do?

Right, cause customers are too stupid to manage their own IPv6 firewalls, it's for their own good! /s
Well, I wouldn't put _any_ service on a public network, unless it is explicitly required. Firewall is all well and good, but security in depth is even better.

Private networking is good. IPv6 doesn't help here at all.

The easiest firewall in the world is one that is set up to deny all traffic from all sources. Which is how any decent firewall is configured by default anyway.

I'm not saying that running a private network doesn't provide genuine security value, only that it drastically complicates your networking architecture for very little security benefit. Organizations can decide whether that trade-off is worth it, for organizations with deep threat models like militaries and banks, it's probably worth it. For 99% of the private sector, it's folly.

"Private networking" as defined by assigning private-range IP addresses are only private as long as there is no route to your network, or as long as it's isolated on a dedicated vlan (even then, there could be some rogue machines).

In the first case, you need a firewall for IPv4 anyway. In the second case, that would also work with IPv6.

Disclaimer: I know nothing about Azure/AWS internals.

This phrasing is really problematic. Using internet addressing (vs ambiguous addresses) does not make your network "public". Just like using unique MAC addresses doesn't. Confusing global addressing with public reachabiliy is exactly the rhetoric used by AWS, Azure etc to scare people into building mazes of ambiguously addressed 10.x networks.
What? Private networks are defined as networks, that use private address ranges[0]. They are most certainly not AWS "rhetoric".

And why are unique MAC addresses a problem?

[0] https://en.wikipedia.org/wiki/Private_network

Private address ranges doesn’t make a network private. Firewall does.

If I know the external, publicly addressable IP address of your router (e.g. 135.77.9.106), and no firewall whatsoever, there’s nothing at all preventing me from doing `ip route add 10.0.0.0/8 135.77.9.106`, and voila, I’d have a route to your “private” network.

Using private addresses vs globally unique offers no security benefiy whatsoever.

I was referring to "public network".

(Though that WP page seems also to have self-coined the "private network" phrase and I don't think it's an estabilished term in this meaning. The first and second references off the leading paragraph talk about "private internets" and "unique local addresses" respectively).

Therefore job security of old school network administrators is the main factor against IPv6 coverage.

Hopefully one of the big cloud providers figures it is in their best interest to have a much bigger address space and make all this busywork sinecure obsolete.

Good luck administering IPv6 networks, they are so much easier to understand.
I’m pretty confident that this statement is mostly true without sarcasm, and that you are in the minority.
How is managing a NAT easier than managing a firewall?
I’m one typo away from accidentally allowing IPv6 access to every machine in my network with my pf config on my home router. (I know this because I’ve done it one time, and didn’t notice for about a week.)

There is no such typo i could make with my single shared public ipv4 address because it’s just one address. Saying “allow” by accident isn’t enough, I’d have to somehow accidentally configure the particular ingress port to NAT to a particular internal machine, and even then it would only affect that machine and no other.

(Full disclosure, i actually like IPv6 and am in full favor of everything moving to it. This is in spite of the above, but i at least recognize that the above is the case.)

I work with Fortune 50s in cloud, and they can barely manage ipv4. If you're in a digital native it's different, but in my experience most behemoths do not inspire confidence with how on top of their network infrastructure they are.
This is a bit like saying “customers can barely manage driving a stick shift with a manual choke — we shouldn’t let them drive automatics!”

IPv6 isn’t amazing, but it makes many of these problems simply disappear. Of course [0] networks should be isolated, but this should be achieved with a firewall that, by default, disallows connections between the public Internet and private networks. And that’s about it — every VM has a globally unique address, routing just works, one company (if permitted) can connect to another company’s endpoints, firewalls can be deployed where they make sense instead of being forced to exist exactly where inconsistently-addressed networks meet, etc.

The entire mess of designing and negotiating allocation of extremely limited IPv4 addresses for private systems simply disappears!

[0] Beyond corp has something to say about this.

I'm not convinced these 2 are related. If AWS wanted, they could start promoting IPv6 as a panacea to all internal network problems (if they had it working, that is), and the competition would have a hard time catching up. My guess is the reason is more mundane: supporting IPv4 is just easier, especially if you take into account the number of their services.
While this is probably exactly the spin the AWS sales team head in mind, it conveniently glosses over all the IPv4 related products customers won’t need anymore if they use IPv6, and thus pay a lot less for their simplified network infrastructure.

AWS has no incentive to support IPv6 from a sales perspective.

In addition, IPv4 addresses are an asset that has monetary value; and there’s no reason AWS would want to drive down the value of the asset by helping customers migrate to IPv6.
Have you heard of the Jevons paradox https://en.wikipedia.org/wiki/Jevons_paradox? It is really insightful. If you make something more efficient/ easier to use, people will use more of it. That is exactly what I think about the switch from IPv4 -> IPv6. Networking is really complicated now and if you happen to make it just a bit easier and cheaper, people will use it and the related services more.

Stateful NAT is a real burden on bigger networks and is at least a chore on smaller networks. It at least doubles the complexity of managing a network especially when you have a DMZ that should be used from some "private" and some "public" endpoints.

Sure there is. AWS has a vested interest in the value of IPv4 going down as much as possible.

Owning IPv4 addresses is a requirement of AWS’s core business. Unless people stop using IPv4, then AWS cannot sell those addresses. There is no incentive for the addresses to increase in value.

Further, if people continue to use IPv4, then AWS has to continue to acquire even more IPv4, and AWS wants the price of those to go down so that acquiring them is cheaper (or wants people to stop using IPv4 so that they can stop spending money on them altogether).

> Unless people stop using IPv4, then AWS cannot sell those addresses. There is no incentive for the addresses to increase in value.

But if people stop using IPv4, the asset (billions of dollars by some accounts) becomes worthless... AWS are passing on the cost for public IPv4 addresses now, so there's even less incentive.

AWS' business model is not speculating on IPv4 addresses. But the fact that smaller providers can't get IPv4 allocations coincidentally works in big cloud providers' favor (and incumbent ISPs). The slower you deploy IPv6 the longer you defer that cost and the longer you enjoy your advantage in address space capacity.
> These provide a strong lock-in effect when customers spend effort to build the complex infrastructure for them in the name of security, even though in reality they hurt security through unnecessary complexity, addressing ambiguity, etc.

How do you hurt security by preventing external access to your internal services?

Was this a jeopardy style reply?

In case not: it's about doing it the wrong way (excess complexity and ambiguity -> hard to understand/analyze/monitor). Using globally addressing is orthogonal to controlling access to your internal services - you can do it using firewalling or various other means. Eg on AWS you get a default-deny firewall.

By making it necessary to use automated tooling and even obtain certifications just to set up everything required for private networking. The complexity of a properly configured service mesh in AWS is staggering, extremely hard for newbies to get right, and easy to fuck up big time down the road. That hurts security.
> By making it necessary to use automated tooling

It isn't.

> and even obtain certifications

No, not really.

> just to set up everything required for private networking.

Frankly, no.

I'm tempted to agree only in one aspect, which is apparently you are completely unfamiliar with the topic, and the degree of confusion you are showing in your comments suggest you would benefit from an introductory course on the topic, or in the very lease a 5-minute read through the service's documentation.

Certification wold only help because you would need to learn the basics to pass those, and learning the basics would be enough to prevent you to fill in the gaps in your understanding with fabricated nonsense.

> The complexity of a properly configured service mesh in AWS is staggering, extremely hard for newbies to get right, and easy to fuck up big time down the road. That hurts security.

People being way over their head because they can't even grasp a FAQ will definitely hurt security, but the root cause of this failure mode is sheer incompetence.

As the saying goes, poor craftsman blame their tools, and here you are with a tool-blaming fest.

Are hundreds of publicly addressable services better?
Yes, because publicly addressable does not mean publicly accessible. Set the ACLs to deny by default.

In the v4 world, one can easily accidentally allow access by inadvertently sending traffic toward the wrong group of colliding addresses or otherwise messing up any of a number of things that ought not to be necessary in the first place.

You need both firewalls and private addresses, of course.

Anything else is amateur hour madness.

Why do I need private addresses?

Okay, in real life I need private addresses because I connect to things that are only available over IPv4. So there’s some negotiation to make sure that my private network does not have an addressing conflict with the other network, there are NATs in the way, and traceroute gives output that is every bit as bad as you would expect. The ACLs that everyone (arguably quite reasonably) sets up suck are fiddly because the clients don’t have well defined address ranges. When people allocate /24 subsets out of IPv4 private space, the probability of collision is annoyingly high. Amateur hour indeed.

I would take globally unique but “private” IPv6 addresses, over private links, with private routes (dynamic or static), and ACLs that actually make sense any day. Heck, I would happily go IPv6 only!

> private addresses

Private addresses offer no security benefit whatsoever. If you have no firewall, nothing at all prevents me from doing `ip route add 10.0.0.0/8 your.routers.ip.here`

You don't need private addresses. You may need non-routable addresses, but they do not need to come from a "private" network range.
Maybe just maybe. Customers don’t really want IPv6 but are forced onto it. Ipv6 is not human usable and gets rid of a bunch of network design norms.
This “human usable” argument gets trotted out so much on here and elsewhere, but the same people would be surprised to know about HTTP/2, TLS and the like, which by that definition, isn’t human usable either because of binary formats and encryption.

People never interact with these protocols directly and use a layer of indirection such as a HTTP/2 client for HTTP, and the same applies for IPv6: use DNS (or your hosts file).

There are a number of relevant issues here, including the general problem that DNS is not trustable and is not reliable or not reliable enough to use for configuring routers and firewalls. It is not even necessarily accessible or usable for reverse lookup at all. DNS wasn't really designed for common cases where network administrators enter IP address prefixes. That could probably fixed to some degree using a name system that was designed for security use, including operation when the network is partitioned or wildly malfunctioning.

And of course the need to maintain two sets of IP addresses and two sets of IP address prefixes - even and especially in DNS itself - is probably the number one factor slowing down the deployment of IPv6. That and far too many places, far too many interfaces, far too many protocols, and far too many APIs (notably Berkeley sockets) that are not transparent to which network layer protocol is being used or what the address format is. The wire format, transfer format, configuration format, and administration of DNS address records is a case in point.

Adding another DNS record or changing a socket listener is hardly the issue though. Most sysadmins are unfamiliar with IPv6 networking concepts such as NDP, DHCPv6 and so on, and having to learn a new system is what hinders its adoption.

Unfortunately, such changes are quite common in networking; Linux networking has many moving parts these days, there was the move to iproute2 and nftables, and the like, so one can only try to best keep up with the changes.

> the same applies for IPv6: use DNS (or your hosts file).

And for reverse DNS, PTR records? What should we use there?

If you're setting up your private DNS resolvers, you can add PTR records to it. There's nothing special about PTR records in IPv6, they're just DNS records for "in6-addr.arpa" instead of "in-addr.arpa".
How many PTR records should you add for the customary 64-bit Interface ID that applies to one device configured with SLAAC?

https://datatracker.ietf.org/doc/html/rfc8501#page-4

Some of these "not human usable" complaints about typing/memorizing/pattern matching IPv6 addresses remind me of how long the distributed version control industry struggled with content-addressed storage and how "human usable" it was or was not. As the legends go Monotone spent years of engineering and lots of complicated code trying to build nice human usable sequence numbers in a distributed fashion, and then git just said "do the simple, stupid thing: show the (prefix of the) hash, people will adapt" and people did.

IPv6 doesn't seem "human usable" sometimes in large part because you aren't actually using it. People adapt. The human skills in pattern matching are robust: there are new tricks to learn, but there were always tricks to learn. (IPv4 addresses aren't "human usable" either if you sit down to truly assess absolutely how many RFCs are involved to build the patterns "everyone" has internalized that seem "easy". They are easy because they are familiar, because you use them often, because you've already adapted to them.)

If you remember the early Internet (the 90's, before NAT took off), you'd realize end-to-end connectivity, globally unique addresses is actually the norm. IPv6 is bringing that back. I remember having public IPv4 on my desktop!
>norms

More like it gets rid of band-aids

Humans don't use IP.

Computers do.

>not internet style architectures with end-to-end addressing.

The inside of my service is not the internet, even though my service may be exposed on the internet; why would I want internal implementation details externalized?

I'm not seeing how security is harmed by making things inaccessible to the public Internet if you don't intend for services you do not control to ever access them directly.