[roland-s]

I'm not too knowledgeable on the subject but I know there are a lot of groups working on persistent storage for Docker. I think most of the time this means a docker volume that lives on some network attached storage (this could be redundant across the compute cluster or on dedicated storage servers), which is then linked to a container via the network. If you remove that container or destroy the host, the volume persists and you can recreate the container on a new host and re-attach it to the associated volume (automatically).

As far as databases and horizontal scaling, this could probably be done currently within each Docker container at the database layer as you mentioned (CockroachDB also comes to mind). So let's say you add a new docker container with its own persistent+redundant storage volumes, and that container is running a cockroachDB node. You can add more containers as cockroachDB nodes, each with their own persistent+redundant storage, and you get all the DB sharding etc across nodes from cockroachDB while the redundancy+persistence are handled by Docker (plus storage solution) as far as restarting, migrating, reconnecting, etc. This probably has all sorts of problems I'm blind to but that's my rudimentary understanding.

Handling horizontal scaling, sharding etc at the storage layer directly through Docker sounds much much harder, and I don't even know what I've been talking about so far so I'll leave it at that

Here's a list of some people doing persistent Docker storage :

  Flocker
  Portworx https://portworx.com/
  StorageOS https://storageos.com/
  Rancher Longhorn https://github.com/rancher/longhorn
  Quobyte https://www.quobyte.com/containers
  Nexenta https://nexenta.com/solutions/containers
  Infinit https://infinit.sh/docker
  EMC REX-ray https://github.com/codedellemc/rexray
  Ceph
  Gluster
  Mesos https://mesosphere.github.io/marathon/docs/external-volumes.html
  Kubernetes https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  vSAN
  Nutanix http://next.nutanix.com/t5/Nutanix-Connect-Blog/Containers-Enter-the-Acropolis-Data-Fabric/ba-p/11122
  Netapp (Docker volume plugins for SolidFire+ONTAP+E-series)
  AWS EBS/EFS