Hacker News new | ask | show | jobs
by NightMKoder 460 days ago
This is actually a fascinatingly complex problem. Some notes about the article: * The 20s delay before shutdown is called “lame duck mode.” As implemented it’s close to good, but not perfect. * When in lame duck mode you should fail the pod’s health check. That way you don’t rely on the ALB controller to remove your pod. Your pod is still serving other requests, but gracefully asking everyone to forget about it. * Make an effort to close http keep-alive connections. This is more important if you’re running another proxy that won’t listen to the health checks above (eg AWS -> Node -> kube-proxy -> pod). Note that you can only do that when a request comes in - but it’s as simple as a Connection: close header on the response. * On a fun note, the new-ish kubernetes graceful node shutdown feature won’t remove your pod readiness when shutting down.
2 comments

With health I presume you mean readiness check. right? Otherwise it will kill the container when the liveness check fails.
By health check, do you mean the kubernetes liveness check? Does that make kube try to kill or restart your container?
More likely they mean "readiness check" - this is the one that removes you from the Kubernetes load balancer service. Liveness check failing does indeed cause the container to restart.
Yes sorry for not qualifying - that’s right. IMO the liveness check is only rarely useful - but I've not really run any bleeding edge services on kube. I assume it’s more useful if you actually working on dangerous code - locking, threading, etc. I’ve mostly only run web apps.
liveness is great for java apps that spend all their time fencing locks. I've seen too many completely deadlock.