Hacker News new | ask | show | jobs
by smarterclayton 3414 days ago
Agree, it would be better if the nodes could not trivially request other secrets. But you can request new containers placed on any node and a root escape that works on one node almost certainly works on the rest.

We usually recommend subdividing the node acls by namespace when running disjoint node sets (where tenant A can't schedule onto tenant B's nodes). More fiddly than it has to be in Kubernetes today.
2 comments
piva00 3414 days ago
We are currently facing the same problem and came to the same conclusion, we are subdividing our node pools by namespace (and using namespacing for multi-tenancy) but it "feels" very convoluted and a smell that there's a feature missing in Kubernetes itself to handle this more graciously.
link
smarterclayton 3413 days ago
The node placement admission controller was definitely intended for that. In combination with a simple controller for allocating namespaces to sets of nodes.

The real solution is of course to limit what nodes can see - I won't say it's trivial, but it's an O(1) check based on the pods scheduled on that node.

link
cpuguy83 3413 days ago
I think in this case you'd already be compromised if you have to worry about a user being able to essentially maliciously place a container. Really a job for RBAC and not the scheduler itself.
link
smarterclayton 3413 days ago
I'm referring mostly to users who are legitimately able to run workloads on a non-zero percentage of the cluster. It's fairly hard to defend against a user with the ability to schedule even a limited amount of workloads, because good scheduling usually spreads (in which case you can compromise more machines and then take them out of service briefly) or packs (in which case you get put on nice target rich environments).

That's partially why we (openshift) just don't allow containers to run as root by default - it sucks for new users (most images assume root) but it dramatically lowers the risk of compromise across the board in any scenario with multi-tenancy. Agree that with isolated nodes per tenant you wouldn't have this issue, which is a more straightforward RBAC story.

link