[sgarland]

> Haven't tried any distributed volume storage yet.

Having tried Gluster, Ceph/Rook, and Longhorn, I strongly recommend Longhorn. Gluster is kinda clunky to setup but works, albeit with very little built-in observability. Ceph on its own also works but has some fairly intense hardware requirements. Ceph with Rook is a nightmare. Longhorn works great (as long as you use ext4 for the underlying filesystem), has good observability, is easy to install and uninstall, and has lower hardware requirements.

Its main drawback is it only supports replication, not erasure coding, which tbf is a large contributor to its ease of use and lower hardware requirements.

[remram]

longhorn has no authentication at the moment, so any workload running in your cluster can send API requests to delete any volume. I think they are working on it but it might not be the best solution unless you deploy a security policy to prevent network access to the API pod.