[kossae]

I can honestly say the only thing we found the full Docker line up (Containers, Swarm, Machine, etc.) good for large scale dev or staging/testing stuff. There are definitely uses for each of these tools, or other combinations, that have been successful for other people. It seems like Docker has a lot to focus on keeping up with the superior orchestration platforms that are out now. This in turn means quickly outdated (or missing) docs, inconsistent behavior of certain features, and other little "gotchas".

Depending on the size of your team, I would recommend Kontena (www.kontena.io) or Kubernetes (https://kubernetes.io/). While I haven't used the latter, a group of two devs is re-architecting our entire system to use Docker and Kontena. Nearing the end, the whole process seemed incredibly fast to iterate. While K8s seems far more proven in real production environments, it seemed much more daunting to us with more features than we needed for our use case.

[raesene9]

I'd recommend people read the "Risks" section of Kubernetes Docs on secrets before using it... https://kubernetes.io/docs/user-guide/secrets/#risks.

secrets stored in plain text and (by default) transmitted in plain text between etcd services, does not fill me with confidence on their security.

[cpitman]

Both of these are easy to fix. Configure etcd with peer certificates for clustering and a cert for client-server connections (https://coreos.com/etcd/docs/latest/v2/security.html). If you need encryption at rest, encrypt the filesystem.

If setting up a secure cluster is daunting, then use a distribution that handles it for you. OpenShift (https://www.openshift.org/) is built on kubernetes, and it's install is secure by default.

Disclaimer: I work for Red Hat, and spend lots of time on OpenShift consulting.

[sandGorgon]

kubernetes is NOT easy to setup in a secure way. I am part of several k8s SIG and we have been discussing these issues for a while now. e. g. https://github.com/kubernetes/kubernetes/issues/27343

there are tons of these niggling issues that are cropping up.

the complexity of using k8s goes up exponentially every day. I have bo doubt it is a great piece of tech.. but at this point, it seems tailor made for consulting.

[benth]

There's still rooted node / kubelet access control to worry about, which does not look to be a problem with Docker. (https://github.com/kubernetes/kubernetes/issues/40476)

[smarterclayton]

A rooted node has access to everything that lands on that node, and anyone who can reproducibly escape to root on a node from a container can do so on any node they can schedule on.

It's definitely something we'll fix in Kubernetes, but rooting workloads is the primary problem, and secondary acl defense in depth is good but won't block most attacks for long.

[bigmac]

There's no way to schedule anything from a worker node -- Swarm follows a push model for all scheduling decisions; worker nodes never pull anything. This is the best ACL model possible: the one that doesn't exist because worker nodes have zero ability to perform actions.

Default ACLs are clearly the most important line of defense in an orchestrator's security model, because whether a container escape can happen is not something the orchestration system has control over.

[smarterclayton]

I'm not sure I disagree, but pull vs push with the same ACL rules in place is the same outcome. A secure Kubernetes configuration would also not be able to schedule from a worker. Partition of secrets is important, but anyone able to trigger node compromise still sees secrets and workloads anywhere they can schedule.

[raesene6]

The thing is with security, default matter. Many people will not change default settings (especially in complex products like Kubernetes), that's why I suggested that people thinking about using Kubernetes should read and be aware of those risks.

On OpenShift I'd be very interested to see a list of the secure defaults that have been chosen, is there a list of the changes from base Docker/Kubernetes policies available anywhere?