[vbezhenar]

Yesterday I was trying to track weird bug. I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.

Google Chrome shows that 6 requests require 2-3 seconds to complete simultaneously. 3 of those requests are tiny static files served by nginx, 3 of those requests are very simple DB queries. Each request completes in few milliseconds using curl, but few seconds in Google Chrome.

Long story short: I wasn't able to track down true source of this obviously wrong behaviour. But I switched ingress-nginx to disable HTTP 2 and with HTTP 1.1 it worked as expected, instantly serving all requests.

I don't know if it's Google Chrome bug or if it's nginx bug. But I learned my lesson: HTTP 1.1 is good enough and higher versions are not stable yet. HTTP 3 is not even supported in ingress-nginx.

[dilyevsky]

> I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.

My guess is it has more to do with resources you probably allocated to your app (especially cpu limit) than any networking overhead which should be negligible in such a trivial setup if done correctly

[vbezhenar]

No, I don't use cpu limits and memory limits are plentiful. Besides, the only difference between working and non-working setups were literally single switch `use-http2` in the ingress controller and it was repeatable experience.

[dilyevsky]

That is indeed interesting. I’ve used nginx with grpc (which uses http2) years ago and definitely did not see any such latency issues even at 50k qps

[mgaunard]

Kubernetes makes everything slow and complicated.

Why do you even need to have proxies or load balancers in between? Another invention of the web cloud people.

[vbezhenar]

That's not my experience. My experience is that kubernetes does not introduce any noticeable performance degradation. My message was about nginx, not kubernetes.

Front TCP load balancer allows to use a single IP address for multiple backend servers. It allows for server maintenance without service interruptions. Or even server crash.

Ingress controller is nginx which does reverse proxying. It handles HTTPS termination and it handles routing to the application which is supposed to serve the request. It's either another nginx instance configured to serve static resources or some kind of HTTP server which implements an actual server logic.

I agree that having two nginxes for static resources seems weird, but that's not a big deal, because nginx does not consume much resources.

[mgaunard]

In real life you cannot achieve that level of failure transparency without specifically building your application for it.

If request N goes to server A, then request N+1 from the same session goes to server B, most likely that request will fail because the session does not exist within that server's context.

[xyzzy_plugh]

nginx is notoriously bad at not-HTTP 1.1. I wouldn't even bother trying.

Envoy is significantly better in this department.

[doublepg23]

Huh TIL. I had always considered nginx the “fast” http server.

[dilyevsky]

Nginx is fast enough for most applications. I wouldn’t bother switching if you dont need the power of “software defined” proxy

[xyzzy_plugh]

Performance aside, if you need fully functional HTTP/2 and maybe HTTP/3 support then you're better off not using nginx. It's a pain.

[apitman]

What were your reasons for moving the site to kubernetes?

[vbezhenar]

Because kubernetes is much easier to operate than dozens of scripts and docker composes thrown around servers. We have like 5 sites, each site operated by like 5-10 services thrown around 4 servers without any redundancy, autoscaling and even consistency. It's truly is a mess.