[mstump]

I've been running a couple of petabytes in production with Docker and Cassandra for a couple years (around 2k nodes). I've rarely seen FS corruption, however I must qualify that this is on bare metal. They could be running into issues with the interaction with EBS? This is more of a screed than an argument backed by specific details and facts.

[tayo42]

How do you handle making updates to configuration with all your nodes in containers? Do you blue green deploy the cluster or something? Run config management in the container?

[mstump]

Configuration comes from the environment. We store the configuration per cluster in a centralized store (C*, etcd, SimpleDB). We bake images that contain everything else.

Depending on the customer and the tech involved we'll do blue-green by doing a controlled rolling push of the config or image after it makes it through the dev/test cycle. Also depending on the type of tech we'll store actual data on network or host volumes.