Hacker News new | ask | show | jobs
by plughs 2395 days ago
Not as far as I can tell, and as far as I'm concerned it is the biggest pain point when debugging k8s problems. If your pods is cycling though CrashLoopBackOff and 'describe' has no useful information and the 'logs' don't give you a clue, you are flat out of luck. That pod is gone before you can extract any useful information from it.

I'd like to see some means of keeping a pod running even if the application is crashing. Sometimes logs are being sent to a file, sometimes running the application manually gives a clue. Lacking that - well I've found various tricks such as changing the image to 'ubuntu' and replacing the command with 'tail -f /dev/null' so a crashing pod will stay alive and I can exec into it and run the real command by hand. But it's a hack and doesn't always get the info you need.
1 comments
oso2k 2390 days ago
If pod logs (`kubectl logs pod-name-hash`) and `kubectl describe` is not giving you useful info, then try checking the Event Logs with `kubectl get events` in the pod's namespace [0].

In OpenShift (I'm a Red Hat Consulting Architect), we have the ability to debug Pods (Failed, Running, or otherwise), DeploymentConfigs (Deployments), some other things, but not BuildConfigs. You just do something likes `oc debug pod-name-hash` or `oc debug deployment/deployment-name` [1]. What this does is start up the Pod with all the configuration (ConfigMaps, Env Vars, Mounts, Secrets, etc.) but replaces CMD/ENTRYPOINT with a shell using `/bin/sh`. This might be magic bits we've added to our `oc/kubectl` (they're one in the same in OpenShift) but I don't see references for similar functionality in Kubernetes. Useful for identifying where a configuration may be slight off what you or CMD/ENTRYPOINT are expecting.

[0] https://kubernetes.io/docs/reference/kubectl/cheatsheet/#vie...

[1] https://cookbook.openshift.org/logging-monitoring-and-debugg...

link
plughs 2389 days ago
`oc` sounds like a thing of great beauty but I'm sure I couldn't convince anyone to pay Red Hat licencing fees.

Our project is a bit unconventional and I'm usually dealing with applications that have been minimally and reluctantly containerized. Pods that crash with no logging and no tracing is probably something I have to deal with more than others.

link
oso2k 2388 days ago
If cost is initially a big issue, checkout the upstream to OpenShift, OKD [0]. Mind you, a Kubernetes distribution for production-ready workload will take some work. This why OpenShift is one of the few Kubernetes distributions to support multiple underlying infrastructure layers (BareMetal, OpenStack, VMware, AWS, GCP, Azure). The Public Cloud providers offer hosted Kubernetes because it makes your life simpler/their life more profitable and the customer’s life simpler. And if OKD interests you, you can play learn in a guided manner in our sandbox clusters at https://learn.openshift.com

[0] https://www.okd.io/

link
oso2k 2388 days ago
Oh there’s also an OpenShift user’s mailing list where people get fairly direct access to knowledgeable Red Hatters and even the devs.
link