[ps]

Reading other comments here, I guess I am lucky enough Ceph still works fine for me, however I am still baffled by its potential performance. Last cluster deployed is 3 nodes each with 4x P4610 NVMe as OSDs, 25Gbps network pushes only 20k IOPS during 4k read, queue depth 128. Single disk is supposed to push around 600k...

“One of the issues is that the officially recommended way to install it seems to be rook-ceph with kubernetes.“

I believe the recommended way is using cephadm.

I have been using ceph-ansible since luminous and had to deal with nasty issues or quirks as well caused probably by complexity and human errors.

[sekh60]

You are correct about cephadm being the recommended install method.

For NVMe OSDs how many daemons are using the same drive? Normally to get performance you have to colocate two or four osd daemons on a single NVMe device.

Also ceph benefits the most from massive parallelism, how many clients do you have? 4 OSDd is not a lot to spread the load over, even if the drives are lightning fast.

There is a new io pathway in the works called crimson which should make NVMe drives more performant. I was hoping a preview would lend in Pacific, but I guess it isn't close enough to being ready.

[rjzzleep]

Cephadm is very new. So yes it may be the recommended solution NOW. Precisely because all previous solutions were very fragile. I don’t have any experience with it for these reasons.

Somehow Linux new filesystems take very long from initial release to reliable use. As seen also with btrfs.

Maybe filesystems are just hard I don’t know but somehow the solutions seem very Hacky.