[ongy]

I suspect this is partially from google's internal 0 trust cluster networking.

I.e. even if the communication is entirely between components inside a k8s (or borg) cluster, it should be authenticated and encrypted.

In this model, there may be a reverse proxy at the edge of the cluster, but the communication between this service and the internal services wouls still be https. With systems like cert-manager it's also incredibly easy to supply every in-cluster process with a certificate form the cluster-internal CA.

-- Googler, not related to this project

[kevinmgranger]

The policy(?) change came ever since "SSL added and removed here ;-)", right?

That's when I remember seeing a broader shift towards app-terminated TLS.

[cyberpunk]

Why wouldn’t you use istio or cilium for this?

[ongy]

This might be me being daft, but I never quite understood the appeal of doing this with istio. OR also partially just due to the timing of when I started to care about things in k8s world. (Rather recently)

My understanding of that model is that the services themselves still just do unauthenticated HTTP, this gets picked up on the client side by a sidecar, packed into mTLS/HTTPS, authed+unpacked on the server sidecar, then passed as plain HTTP to the server itself.

This is great when we have intra-host vulnerabilities I guess. But doesn't allow to e.g. have code sanitizers that are strict on using TLS properly (google does this).

And while it is a real gain over simple unauthed with untrusted network between nodes, with cilium taking care of the intra-node networking being secure, I don't quite see how the added layer is more useful than using network policies strictly.

(besides some edge cases where it's used to further apply internal authorization based on the identity. Though that trusts the "network" (istio) again.)

[cyberpunk]

For us it’s compliance related first rather than any real security upgrade; we must use mtls between all services (finance) and it’s simply less to manage to use a service mesh.

The cloud provider could read the memory of a k8s node and in theory capture the session keys of two workloads on the same node, and we can’t really protect against that without something like confidential computing.

We get some other benefits for free by using istio though like nice logs, easily sending a chunk of traffic to a new release before rolling it everywhere, or doing fully transparent oauth even in services which don’t support it (oauth2proxy and istio looks at the jwts etc).

[liveoneggs]

in the modern world extra network hops, novel userland network stacks, and additional cycles of decrypted/re-encrypting traffic make your apps go faster, not slower.

[nine_k]

Not sure if it's ironic or not. Because it should be not.

AES-NI gives you encryption at the speed of memcpy basically. Userland network stacks are faster because they don't incur the kernel call cost. With that, if your NIC drivers support zero-copy access, an extra hop to a machine in the same rack over a 10G link is barely noticeable, may be shorter than an L3 miss.

The cost of this is mostly more hardware and more power used, but not much or any additional latency.

[liveoneggs]

lol

[grogenaut]

Why add another layer if you aren't already using istio or cilium?

[cyberpunk]

Because it’s zero configuration auto mtls between all the services in your cluster (or intra-node if cillium) instead of managing a tls cert for every service?

[sofixa]

Zero to little configuration at point of use, but a lot of upfront configuration, maintenance, fun issues when you need something slightly less traditional (e.g. something that needs raw TCP or heavens forbid, UDP). Different trade offs for different situations.

[cyberpunk]

I still think it’s far less than managing tls per service.

Every component needs a different tls configuration, vs one time installing istio.

Raw TCP is supported by istio even with mtls, you just have to match in your VirtualServices on SNI instead of Host header.

We routinely mix tcp and http services on the same external ports, with mtls for both.

UDP I don’t really see how is relevant to a conversation on tls

[sofixa]

> one time installing istio

And never update it afterwards?

> UDP I don’t really see how is relevant to a conversation on tls

You might have UDP services alongside your TCP/HTTP behind TLS.