[jppittma]

How is the kubernetes secret API lock in? Genuinely wondering - were you trying to use that deployment yaml for something other than a kubernetes deployment? For most applications, you should be mounting the secret on your application, then you can inject it as either an environment variable or a json file that your application reads in an environment agnostic way.

Then, on the backend, you can configure etcd to use whatever KMS provider you like for encryption.

[stackskipton]

Because you can't run the container, even for development outside Kubernetes.

Yes, you can mount Secrets as Volumes or Env Var in Kubernetes which is fine but I'm not talking about "How you get env var/secret" but "Methods of dealing with config."

[jppittma]

Yes you can? The container should be completely agnostic to the fact that it's running in kubernetes. You can do config the same way. Configmaps are mounted as regular files and environment variables. The application doesn't care if the configmap came from the cluster resource or a file your created on your dev machine with dev credentials. You can mount local files into the container yourself. It's docker run -v "source:destination" I think.

[joshribakoff]

One of you is talking about mapping a secret to an environment variable and the other one of you is talking about having the work load make an API call to retrieve the secret. You’re not even talking about the same thing.

[jppittma]

The k8s api server is the thing that's configured to talk to your Thales or whatever. On managed kubernetes, these are usually preconfigured to talk to the vendor -- that's the difference between a secret and a config map. The secret is encrypted when it's stored in etcd.

You'd be forgiven for being mistaken however, because this encryption is handled in a way that's transparent to the application.

If you're talking about your application making a call to the k8s api server, then you shouldn't do that unless you're developing a plugin. The kubelet knows how to retrieve and mount secrets from the k8s api server and display them as environment variables to the application. You just declare it as a part of your deployment in the podspec.

[stackskipton]

sigh I’m extremely competent Ops type and I know. If you mount secrets as Volume or Env Var, that’s Config file or Env var from Application PoV. We are looking at this from Application PoV.

I’ve seen Applications that do direct calls to Kubernetes API and retrieve the secret from it. So they have custom role with bindings and service account and Kubernetes client libraries.

[jppittma]

If you're not developing k8s operators, you're calling the api server directly, then complaining about lock in, then that's a skill issue. If you're developing k8s operators, then you should use a tool like kind for integration tests and dependency injection for other stuff and the concept of lock in doesn't make sense. You can also deploy your helm chart directly to kind.

[cassianoleal]

Don’t use live system secrets and credentials when running your application locally. Then you don’t need to access the same secrets.

Keep it simple and design your applications so they’re agnostic to that fact.

It’s really not that hard, I’ve been doing this for at least 6 or 7 years. A little bit of good engineering goes a long way!

[Nilocshot]

This is where I like things like Tilt. If you're deploying to a k8s cluster, it's probably a good idea to do local dev in as close to a similar environment as possible.

Bit more of an initial hurdle than "just run the docker image"; however.

[stackskipton]

I've look at Tilt and it's another abstraction for Kubernetes which rarely ends well at scale.

However, most of time, Devs don't need to develop on Kubernetes since it's just Container Runtime and Networking Layer they don't care about. They run container, they connect to HTTP endpoint to talk to other containers, they are happy. Details are left to us Ops people.

[fragmede]

It seems contradictory to say that Tilt is an abstraction over kubernetes and say that won't work at scale, but then volunteer ops to be a layer of abstraction over kubernetes as a solution.

FWIW, Skaffold.dev is similar to Tilt, and has been working out great. "skaffold dev" on the cli or the corresponding button in the users IDE starts up a local kube cluster (minikube), the dev code in a container, any other configured containers, optionally opening a port to attach a debugger, and watches the code for changes and restarts the container with the dev code when there's changes. Developers aren't beholder to the capacity of whoever's on call on the ops team to manage the containers they need to be productive. The details of pods and CRDs and helm and ArgoCD and kubectl are abstracted away for them. They just run "skaffold dev" and edit code. Ops gets to run "skaffold render" to build and tag images, and generate the corresponding kubernetes manifest.

[stackskipton]

Ops is not a layer of abstraction over Kubernetes no more then Dev is layer of abstraction over Python. We both have different responsibilities and thinking that Ops is just missing one more library is why it goes so wrong.

Kubernetes is massive beast and I get it. It feels extremely overcomplicated for "Please for the love of all that holy, just run this container." However, trying to abstract away such complexity is like trying to use Golang with some Python to Golang cross compiler. It works until it you need some feature and then oh god, all hell breaks loose.

I have not played with scaffold either but I will say. scaffold render should not be Ops job, I find it goes best when Devs present artifact they believe is ready for production and I can slot into the system. Otherwise, the friction between Devs handing Ops what they think is possibly buildable artifact quickly becomes untenable.