Hacker News new | ask | show | jobs
by BitPirate 878 days ago
It's a bit like drag racing. If all you care about is the performance of a single transfer that doesn't have to deal with packet loss etc, HTTP/1 will win.
2 comments

Yesterday I was trying to track weird bug. I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.

Google Chrome shows that 6 requests require 2-3 seconds to complete simultaneously. 3 of those requests are tiny static files served by nginx, 3 of those requests are very simple DB queries. Each request completes in few milliseconds using curl, but few seconds in Google Chrome.

Long story short: I wasn't able to track down true source of this obviously wrong behaviour. But I switched ingress-nginx to disable HTTP 2 and with HTTP 1.1 it worked as expected, instantly serving all requests.

I don't know if it's Google Chrome bug or if it's nginx bug. But I learned my lesson: HTTP 1.1 is good enough and higher versions are not stable yet. HTTP 3 is not even supported in ingress-nginx.

> I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.

My guess is it has more to do with resources you probably allocated to your app (especially cpu limit) than any networking overhead which should be negligible in such a trivial setup if done correctly

No, I don't use cpu limits and memory limits are plentiful. Besides, the only difference between working and non-working setups were literally single switch `use-http2` in the ingress controller and it was repeatable experience.
That is indeed interesting. I’ve used nginx with grpc (which uses http2) years ago and definitely did not see any such latency issues even at 50k qps
Kubernetes makes everything slow and complicated.

Why do you even need to have proxies or load balancers in between? Another invention of the web cloud people.

That's not my experience. My experience is that kubernetes does not introduce any noticeable performance degradation. My message was about nginx, not kubernetes.

Front TCP load balancer allows to use a single IP address for multiple backend servers. It allows for server maintenance without service interruptions. Or even server crash.

Ingress controller is nginx which does reverse proxying. It handles HTTPS termination and it handles routing to the application which is supposed to serve the request. It's either another nginx instance configured to serve static resources or some kind of HTTP server which implements an actual server logic.

I agree that having two nginxes for static resources seems weird, but that's not a big deal, because nginx does not consume much resources.

In real life you cannot achieve that level of failure transparency without specifically building your application for it.

If request N goes to server A, then request N+1 from the same session goes to server B, most likely that request will fail because the session does not exist within that server's context.

nginx is notoriously bad at not-HTTP 1.1. I wouldn't even bother trying.

Envoy is significantly better in this department.

Huh TIL. I had always considered nginx the “fast” http server.
Nginx is fast enough for most applications. I wouldn’t bother switching if you dont need the power of “software defined” proxy
Performance aside, if you need fully functional HTTP/2 and maybe HTTP/3 support then you're better off not using nginx. It's a pain.
What were your reasons for moving the site to kubernetes?
Because kubernetes is much easier to operate than dozens of scripts and docker composes thrown around servers. We have like 5 sites, each site operated by like 5-10 services thrown around 4 servers without any redundancy, autoscaling and even consistency. It's truly is a mess.
It runs over TCP, you don't need to deal with packet loss.
What they’re suggesting is that under packet loss conditions QUIC will outperform TCP due to head of line blocking (particularly when there are multiple assets to fetch). Both TCP and QUIC abstract away packet loss but they have different performance characteristics under those conditions.
HTTP/1 doesn't have head of line blocking, only HTTP/2 does.
HTTP/1 has parallelism limitations due to number of simultaneous connections (both in terms of browser and server). HTTP/2 lets you retrieve multiple resources over the same connection improving parallelism but has head of line problems when it does so. HTTP/3 based on Quic solves parallelism and head of line blocking.
The only way the first part of your sentence is correct means that the second part is wrong. HTTP 1 pipelining suffers from head of line blocking just as badly (arguably worse) and the only workaround was opening multiple connections which HTTP/2 also supports.
Most browsers only support very limited number of connections so it kinda does
Limitations of certain implementations are irrelevant.

The protocol does not have any such limitation.

Totally, after all it’s not like we live in a real world where these things matter