[miohtama]

QUIC does not work very well for use cases like machine-to-machine traffic. However most of traffic in Internet today is from mobile phones to servers and it is were QUIC and HTTP 3 shine.

For other use cases we can keep using TCP.

[kldx]

Let me try providing a different perspective based on experience. QUIC works amazingly well for _some_ kinds of machine to machine traffic.

ssh3, based on QUIC is quicker at dropping into a shell compared to ssh. The latency difference was clearly visible.

QUIC with the unreliable dgram extension is also a great way to implement port forwarding over ssh. Tunneling one reliable transport over another hides the packer losses in the upper layer.

[exDM69]

The article that GP posted was specifically about throughput over a high speed connection inside a data center.

It was not about latency.

In my opinion, the lessons that one can draw from this article should not be applied for use cases that are not about maximum throughput inside a data center.

[thickice]

Why doesn't QUIC work well for machine-to-machine traffic ? Is it due to the lack of offloads/optimizations for TCP and machine-to-machine traffic tend to me high volume/high rate ?

[yello_downunder]

QUIC would work okay, but not really have many advantages for machine-to-machine traffic. Machine-to-machine you tend to have long-lived connections over a pretty good network. In this situation TCP already works well and is currently handled better in the kernel. Eventually QUIC will probably be just as good for TCP in this use case, but we're not there yet.

[jabart]

You still have latency, legacy window sizes, and packet schedulers to deal with.

[spwa4]

But that is the huge advantage of QUIC. It does NOT totally outcompete TCP traffic on links (we already have bittorrent over udp for that purpose). They redesigned the protocol 5 times or so to achieve that.

[extropy]

The NAT firewalls do not like P2P UDP traffic. Majoritoy of the routers lack the smarts to passtrough QUIC correctly, they need to treat it the same as TCP essentially.

[beeflet]

NAT is the devil. bring on the IPoc4lypse

[hdgvhicv]

Nat is massively useful for all sorts of reasons which has nothing to do with ip limitations.

[beeflet]

sounds great but it fucks up P2P in residential connections, where it is mostly used due to ipv4 address conservation. You can still have nat in IPv6 but hopefully I won't have to deal with it

[mightyham]

In practice, P2P over ipv6 is totally screwed because there are no widely supported protocols for dynamic firewall pinholing (allowing inbound traffic) on home routers, whereas dynamic ipv4 NAT configuration via UPnP is very popular and used by many applications.

[paulddraper]

The NAT RPC talks purely about IP exhaustion.

What do you have in mind.

[skissane]

Why run your K8S cluster on IPv6 when IPv4 with 10.0.0.0/8 works perfectly with less hassle? You can always support IPv6 at the perimeter for ingress/egress. If your cluster is so big it can’t fit in 10.0.0.0/8, maybe the right answer is multiple smaller clusters-your service mesh (e.g. istio) can route inter-cluster traffic just based on names, not IPs.

And if 10.0.0.0/8 is not enough, there is always the old Class E, 240.0.0.0/4 - likely never going to be acceptable for use on the public Internet, but growing use as an additional private IPv4 address range - that gives you over 200 million more IPv4 addresses

[Iwan-Zotow]

Kubes

[unethical_ban]

Rather, NAT is a bandage for all sorts of reasons besides IP exhaustion.

Example: Janky way to get return routing for traffic when you don't control enterprise routes.

Source: FW engineer

[hdgvhicv]

Sure. When I can bgp advertise my laptop with my phone provider and have it update is a second or so globally when I move from tethering to wifi, or one network to another.

No doubt you think I should simply renumber all my VMs every time that happens, breaking internal connections. Or perhaps run a completely separate addrsssing in each vm in parallel and make sure each vm knows which connection to use. Perhaps the vms peer with my laptop and then the laptop decides what to push out which way via localprefs, as paths etc. that sounds so much simpler than a simple masquerade.

What happens when I want vm1 out of connection A, vm 3 out of connection B, vm 4-7 out of connection C. Then I want to change them quickly and easily. I’m balancing outbound and inbound rules, reaching for communities, and causing bgp dampening all over the place.

What when they aren’t VMs but instead physical devices. My $40 mifi is now processing the entire DFZ routing table?

What happens when I want a single physical device like a tv to contact one service via connection 1 and another via connection 2 but the device doesn’t support multiple routing tables or selection of that. What if it does support it but I just want to be able to shift my ssh sessions to a low latency higher loss link but keep my streaming ups on the high latency no loss link.

All this is trivial with nat. Now sure I can use NAT66, and do a 1:1 natting (no PAT here), but then I’m using nat and that breaks the ipv6 cult that believes translating network addresses is useless.

[mort96]

NAT isn't dead with IPv6. ISPs assigning a /128 to your residential network is a thing.

[immibis]

No it isn't unless they want to ban you from using iPhones.

[mort96]

What do you mean? If the v6 configuration is incompatible with iPhones, the iPhone will just use v4

[johncolanduoni]

QUIC isn’t generally P2P though. Browsers don’t support NAT traversal for it.

[dan-robertson]

I think basically there is currently a lot of overhead and, when you control the network more and everything is more reliable, you can make tcp work better.

[m00x]

It's explained in the reddit thread. Most of it is because you have to handle a ton of what TCP does in userland.

[exabrial]

For starters, why encrypt something literally in the same datacenter 6 feet away? Add significant latency and processing overhead.

[sleepydog]

Encryption gets you data integrity "for free". If a bit is flipped by faulty hardware, the packet won't decrypt. TCP checksums are not good enough for catching corruption in many cases.

[lll-o-lll]

Interesting. When I read this I was thinking “that can’t be right, the whole internet relies on tcp being “reliable”. But it is right; https://dl.acm.org/doi/10.1145/347059.347561. It might be rare, but an unencrypted RPC packet might accidentally set that “go nuclear” bit. ECC memory is not enough people! Encrypt your traffic for data integrity!

[mschuster91]

Because any random machine in the same datacenter and network segment might be compromised and do stuff like running ARP spoofing attacks. Cisco alone has had so many vendor-provided backdoors cropping up that I wouldn't trust anything in a data center with Cisco gear.

[qingcharles]

Back in the 90s I discovered the CTO of a major telecoms company was packet sniffing EFnet traffic in one of their datacenters in order to log all the PRIVMSGs to extort a couple of people. Security is only as good as its weakest leak, and all that.

[subscribed]

Ummm, no, The network is completely isolated. No one enters the cage and just plugs something into my switches/routers.

Any communication between the cage and the outside world is through the cross-connects.

Unless it's some state-adversary, no one taps us like this. This is not a shared hosting. No one runs serious workloads like this.

"Unserious"? Sure, everything is encrypted p2p.

[mschuster91]

> No one enters the cage and just plugs something into my switches/routers.

I'm not talking about someone plugging something in. I'm talking about someone pwning your VPN endpoint or firewall, and laterally moving from there. There's always a way to move around unless you are really, really careful (and even that is not enough if the adversary has an exploit for something really deep in the network stack).

At the very least, choose different vendors for your VPN/frontend firewall gear and the rest of your system. That way, an adversary can't just go and pwn every little piece of your network infrastructure with a single exploit.

[subscribed]

This is a very fair point, admittedly, considering multiple vulnerabilities in Juniper and Cisco devices.

I concede my point, thanks for that

[lll-o-lll]

To stop or slow down the attacker who is inside your network and trying to move horizontally? Isn’t this the principle of defense in depth?

[20k]

Because the NSA actively intercepts that traffic. There's a reason why encryption is non optional

[Karrot_Kream]

To me this seems outlandish (e.g. if you're part of PRISM you know what's happening and you're forced to comply.) But to think through this threat model, you're worried that the NSA will tap intra-DC traffic but not that it will try to install software or hardware on your hosts to spy traffic at the NIC level? I guess it would be harder to intercept and untangle traffic at the NIC level than intra-DC, but I'm not sure?

[viraptor]

> you're worried that the NSA will tap intra-DC traffic but not that it will try to install software or hardware on your hosts

It doesn't have to be one or the other. We've known for over a decade that the traffic between DCs was tapped https://www.theguardian.com/technology/2013/oct/30/google-re... Extending that to intra-DC wouldn't be surprising at all.

Meanwhile backdoored chips and firmware attacks are a constant worry and shouldn't be discounted regardless of the first point.

[heavyset_go]

> (e.g. if you're part of PRISM you know what's happening and you're forced to comply.)

Only a handful of people need to know what happens in Room 641A, and they're compelled or otherwise incentivized not to let anyone else know.

[codedokode]

> you're worried that the NSA will tap intra-DC traffic but not that it will try to install software or hardware on your hosts to spy traffic at the NIC level

It might not be able to, if you use secure boot and your server is locked in a cage.

[adgjlsfhk1]

The difference between tapping intra-DC and in computer spying is that in computer spying is much more likely to get caught and much less easily able to get data out. There's a pretty big difference between software/hardware weaknesses that require specific targeting to exploit and passive scooping everything up and scanning

[cherryteastain]

If you are concerned about this, how do you think you could protect against AWS etc allowing NSA to snoop on you from the hypervisor level?

[heavyset_go]

Assuming the PSP isn't backdoored, using AMD SME and SEV theoretically allow you to run VMs that are encrypted such that, even at the hypervisor level, you can't read code or data from the VM.

[codedokode]

You cannot assume that. The solution is to have a server on your territory and use the datacenter only to forward the packets.

[exabrial]

Imaginary problems are the funnest to solve.

[20k]

Its a stone cold fact that the NSA does this, it was part of the snowden revelations. Don't spread FUD about security, its important

[switchbak]

Service meshes often encrypt traffic that may be running on the same physical host. Your security policy may simply require this.

[mort96]

I don't understand what you mean by "machine-to-machine" if a phone (a machine) talking to a server (a machine) is not machine-to-machine.

[szszrk]

I hope you don't think that user-to-machine means that I have to stick my finger in a network switch? :)

Machine-to-machine is usually meant as traffic where neither of the sides is the client device (desktop, mobile etc). Often not initiated by user, but that's debatable.

I would say an server making a sync of database to passive node is machine-to-machine, while a user connection from his browser to webserver is not.