[praseodym]

Actually it is already possible to create persistent disks through Kubernetes by creating a persistent volume claim (pvc) which triggers an actual persistent volume to be created. Cloud provider controllers take care of this, just like they create load balancers for services.

Also there’s already work being done on supporting cluster management from Kubernetes itself: https://github.com/kubernetes-sigs/cluster-api

[lobster_johnson]

That's not what I meant, though. A Kubernetes PV is an abstraction that causes a controller allocate a GCE disk (which of course is another abstraction), but if you create a GCE disk through the CLI/UI/API, then you don't get a PV.

[colemickens]

I think the point is that I don't have to touch the cloud provider's control plane in 99% of cases. Which brings the portability that first attracted me to Kubernetes.

[yebyen]

The portability is not as attractive as a fully integrated solution though, for types of people that are not already attracted to Kubernetes. Where I work, we use AWS and say that "we love vendor lock-in." The more integration with the rest of the vendor's cloud platform, the better.

I think it's a great example to say it is idiosyncratic that I can create a PD by creating a PV, but I can't reciprocally enumerate PDs by listing PVs. Another: Kubernetes secrets are hidden inside K8S and they are not generally accessible through any vendor-provided secrets managers outside of K8S.

I love Ingress, because it allows me to create new hostname-driven application endpoints inside of my cluster without any permissions on the AWS side. The cluster has a load balancer with a star certificate attached to it, the DNS has a star record similarly, and I can create new Ingress records without reaching outside of the cluster.

My InfoSec department hates stuff like that! Now if I use Kubernetes that means they have to learn Kubernetes too, and maybe even build out additional tooling around it, because their AWS-based automated service inventories are not reliable anymore.

I don't really have those star records (I'd really like to have them, it would be really convenient for me as a developer, so I don't have to ask for permission and go through layers of approval when I need to do stupid things, like my job...)

The unfortunate reality that I've encountered is that those teams are more likely to see the encapsulation features like this Ingress configuration as a pure security risk with no benefit to them, plain and simple.

[outworlder]

I think I'm missing something.

K8s needs a role with the correct IAM permissions to create the resources on the AWS side, it can't do it magically. Which means that someone granted these permissions before.

If anything, your InfoSec department should love that. IAM is pretty fine-grained, so they can control what gets created. Types of resources, and whatnot. Just don't grant star permissions to all of EC2.

[yebyen]

(Spoiler alert: you got me, InfoSec typically does love this stuff. If it was just up to InfoSec, these types of things would be approved because they are so well organized and easy to understand. The alternative being, to deal with another vendor that has another completely different security model, which InfoSec actually has to do all the time.)

Fact is InfoSec is not at liberty to set priorities for new technology development. I have to either drag the whole group with me at an institutional pace, or get someone at a higher pay grade "who will stake their career on it," in order to help me meet my goals, which really may or may not be achievable simply by means of adopting this advancement. On the other hand, lots of people who are not InfoSec like to "help" with the ostensible jobs of InfoSec, ...

which is often seen as simply delaying progress until the problem can be fully understood and the solution agreed to by a quorum? Man I can't wait that long

I'm a bit salty can you tell

[outworlder]

You can add a PV manually to tie to some resource that was already created, but you don't get the bi-directional link. I'm not sure I'd even want K8s adding PV's for all volumes (maybe filtering by labels/tags would be ok).

This is approaching Terraform levels of complexity though. Unless you mean that cloud providers should do that when you create a volume, in which case I understand where you are coming from and well, they could do that. Although I'd much have them exposing a native K8s interface to take care of this PV and PVC for me and not showing me the underlying volumes at all.

[manigandham]

Why would you expect that? Or use it that way? The clouds are IaaS, and Kubernetes is like a virtual human that creates resources using the same APIs you would. It's on K8S to keep track of what it's creating.

[lobster_johnson]

That's what I mean by "Kubernetes first".

Why not use Kubernetes to declaratively define nodes, disks, load balancers, firewall rules, DNS rules...? If you look at Terraform, for example, it's just an "object store" (the state file) that you modify in order to "converge" your infrastructure.

If you're already using Kubernetes via GKE or a similar IaaS, then of course Kubernetes has underlying primitives (GCE disks, GLBCs, etc.) that are exposed. But those primitives are exposed because GKE runs on top of GCP.

What I'm proposing is a "KaaS" where Kubernetes is the control plane, without the baggage of an additional API.

[manigandham]

That's because Kubernetes isn't an infrastructure system, it's an application system. More PaaS than IaaS, so you're supposed to define what your app needs and let it do the work.

You don't specify a node but the CPU/RAM your application requires and let K8S scale up the cluster if necessary. Likewise, you specifiy the storage needed instead of managing disks, and set network policies to control load balancing and routing. If you want to live in Kubernetes only today, you can absolutely do that. Launch your cluster once and then use kubectl and yaml for the rest.