[higels]

I just mount each SSD as its own XFS filesystem and use NGINX’s split feature to fill them up.

Not resilient on a system level, but refilling the cache is cheap.

ZFS was generally pleasant from an operability viewpoint once we ironed out the quirks, but the perf hit from no sendfile was too much.

[m11r]

That’s a good point; I never looked at what the perf impact of disabling sendfile would be on even a moderately-loaded webserver.

It’d be really nice to see that fixed like the recent DIRECT_IO additions.

[Quekid5]

Of course the people around these parts tend to have very particular needs and use cases, but for anything resembling the "common case" the performance impact of not using sendfile should be negligible.

(I'll just point of that using sendfile means that traffic is unencrypted... which is probably fine on an internal network, but I've started adopting the stance that even internal network traffic should be encrypted unless there's a very good reason not to do that. An absolute requirement for performance might be a good reason.)

[atomt]

If nginx decided to support ktls they could use sendfile for encrypted traffic as well. Unsure if it is worth it just to make sendfile work however.

[m11r]

I was going to mention kernel TLS hopefully enabling sendfile for mostly-HTTPS workloads, as that’s the direction everything is heading anyway, and without it we don’t get zero-copy for those connections.

Now I’m more curious about the actual threshold where not having sendfile begins causing noticeable performance problems… at what point before you become Netflix?

[namibj]

If your cache can face-tank a HTTP-DDoS, you don't need fragile fingerprinting techniques to distinguish bad from good, thus reducing the user impact (less accidentally-blocked users). The less cost you have for filling that 100 Gbit NIC with your TLS cache traffic, the more boxes you can afford. Internet exchanges are surprisingly cheap to connect to.

Of course sharing resources between a couple services would be good, as NICs and switch ports are sill a way from free.

[shanemhansen]

What about http2?