[totallyunknown]

A german company building an app for watching linear TV. Netflix is actually serving 400Gbit/s per node and already have 800Gbit/s ready.

I think we can scale our setup up to 200 Gbit/s but we are too small. Total traffic is ~2 Tbit/s.

Most challenging is the missing support of QUIC/http3 and KTLS in Golang. Also 100G NIC supply chain is difficult. We use NVIDIA Connect-X 6, but it's impossible to get a version with TLS offloading.

[BonoboIO]

Es liegt mir auf der Zunge welche Firma das ist.

I think it starts with a Wa ... you don’t have to say. I kind of remember to have been stumbled on a Twitter engineering ipv6 tweet. Maybe I m wrong.

For me it’s impressive to get so much data through a computer. But I have one question, what does count as a node, is a node like 1 machine with dual sockets, a lot of ram and a lot of nics or is it like multiple machines combined that act as 1 node like a whole 19 inch rack.

[totallyunknown]

A single node is single socket AMD EPCY node with 32 cores, 64 or 128GB RAM, 1x NIC with 2x100 GB/s. Ja, es ist waipu.tv.

[ddorian43]

What about the disks?

Edit: or linear TV means the recent stream can be all in ram?

[totallyunknown]

All in memory.

[otbutz]

There is quic-go[1] but i don't think that it's sufficiently optimized[2-4] to be used for this kind of workload. caddy will use it to provide HTTP/3 by default[5] in the upcoming 2.6.0 release.

[1]: https://github.com/lucas-clemente/quic-go [2]: https://github.com/lucas-clemente/quic-go/issues/2877 [3]: https://github.com/lucas-clemente/quic-go/issues/2607 [4]: https://github.com/lucas-clemente/quic-go/issues/341 [5]: https://github.com/caddyserver/caddy/pull/4707

[lossolo]

Interesting, do you do a lot of processing in Golang or basically you just use it as a wrapper around sendfile[1] ?

1. https://man7.org/linux/man-pages/man2/sendfile.2.html

[totallyunknown]

We can't use sendfile since all the traffic is HTTPS. Sendfile can only be used if we can use KTLS. Internally we have some kind of cache hierarchy for video objects, these files are transferred out via Sendfile, but the performance gain here is negligible.

[costela]

and yes: the proxy does have some intelligence around "just" forwarding requests. Mostly caching related, but also some upstream routing decisions and a ton of observability.