[Giefo6ah]

Ceph solves the distributed consistent block storage problem very well. But I hardly ever need that problem solved, it's way more often that I need a distributed highly available blob storage, and Ceph makes the wrong tradeoffs for this task.

[q3k]

Ceph is fundamentally an object storage system. RBD (block devices) are built on top of that layer.

[yencabulator]

Yeah. 100% this. Remember, Ceph's storage nodes are called OSD, as in Object Storage Daemon.

The biggest reasons to not use Ceph are:

- You plan on using <=30 disks on <=3 computers. Some of the petabyte scale stuff is just in the way at that scale. Even more so if you're looking at a single computer.

- You don't have at least fractional ops staff that will look at a dashboard regularly. In this world, you're better off using a cloud service.

- You're 100% satisfied with the S3 interface and will never want anything else than its write-once objects. Ceph's writable object support won't gain you anything in that world, and genuinely makes the distributed systems problems involved much harder. Ceph was architected as a distributed filesystem and excels as a networked block store for virtual machines. Ceph can do S3-compatibility, but another implementation can cut off a big chunk of the functionality and provide just S3-compat, and simpler can be better.

I doubt many software projects mentioned in this conversation have gone through the extensive stress testing with glitching nodes while trying to maintain performance that Ceph has; simple systems can be too simple. Ceph is quite well battle-hardened by now. Ceph's dedicated QA hardware pool is likely bigger than many competing projects have tried as a cluster size!

Disclaimer: ex-Ceph-developer.