How do non running utilities increase attack surface? If you're able to execute inside the container couldn't you just write whatever utilities you want in?
There are many exploits that may give one the ability to execute shell commands. If there is no shell or commands to even write a file in the first place, mobility becomes limited.
once in awhile i use 'kubectl exec ' to run some commands against my running prod containers to debug something or extract an environment variable. can i still do that without a shell or anything?
You can’t, no. One approach is a “debug sidecar” container in the Pod that has your desired tools. Linkerd, for example, will add such a container if an annotation is present on the Pod: https://linkerd.io/2.11/tasks/using-the-debug-container/
I usually just include a shell, but to each their own.
Dev tools like bash, ls, grep, etc, have no place in production and only increase attack surface.