[totallyunknown]

We did the same. We've replaced nginx/lua with a cache server (for video) written in Golang - now serving up to 100 Gbit/s per node. It's more CPU and memory efficient and completely tailored to our needs. We are happy that we moved away from nginx.

[BonoboIO]

Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.

[totallyunknown]

A german company building an app for watching linear TV. Netflix is actually serving 400Gbit/s per node and already have 800Gbit/s ready.

I think we can scale our setup up to 200 Gbit/s but we are too small. Total traffic is ~2 Tbit/s.

Most challenging is the missing support of QUIC/http3 and KTLS in Golang. Also 100G NIC supply chain is difficult. We use NVIDIA Connect-X 6, but it's impossible to get a version with TLS offloading.

[BonoboIO]

Es liegt mir auf der Zunge welche Firma das ist.

I think it starts with a Wa ... you don’t have to say. I kind of remember to have been stumbled on a Twitter engineering ipv6 tweet. Maybe I m wrong.

For me it’s impressive to get so much data through a computer. But I have one question, what does count as a node, is a node like 1 machine with dual sockets, a lot of ram and a lot of nics or is it like multiple machines combined that act as 1 node like a whole 19 inch rack.

[totallyunknown]

A single node is single socket AMD EPCY node with 32 cores, 64 or 128GB RAM, 1x NIC with 2x100 GB/s. Ja, es ist waipu.tv.

[ddorian43]

What about the disks?

Edit: or linear TV means the recent stream can be all in ram?

[totallyunknown]

All in memory.

[otbutz]

There is quic-go[1] but i don't think that it's sufficiently optimized[2-4] to be used for this kind of workload. caddy will use it to provide HTTP/3 by default[5] in the upcoming 2.6.0 release.

[1]: https://github.com/lucas-clemente/quic-go [2]: https://github.com/lucas-clemente/quic-go/issues/2877 [3]: https://github.com/lucas-clemente/quic-go/issues/2607 [4]: https://github.com/lucas-clemente/quic-go/issues/341 [5]: https://github.com/caddyserver/caddy/pull/4707

[lossolo]

Interesting, do you do a lot of processing in Golang or basically you just use it as a wrapper around sendfile[1] ?

1. https://man7.org/linux/man-pages/man2/sendfile.2.html

[totallyunknown]

We can't use sendfile since all the traffic is HTTPS. Sendfile can only be used if we can use KTLS. Internally we have some kind of cache hierarchy for video objects, these files are transferred out via Sendfile, but the performance gain here is negligible.

[costela]

and yes: the proxy does have some intelligence around "just" forwarding requests. Mostly caching related, but also some upstream routing decisions and a ton of observability.

[trillic]

100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.

[zamadatix]

40 Mbps seems roughly 5x-10x the bitrate I'd expect for this kind of thing.

[totallyunknown]

100 Gbit/s / 5000 KiB/s is 20000.

[ImprobableTruth]

KiB = Kibibyte, not Kibibit

[mplewis]

“Only” :)

[988747]

Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.

[nly]

It's almost as if on-demand video streaming isnt suitable for broadcasting live events.