[lillecarl]

How do people running Ceph and other exotic filesystems deal with performance? What performance is considered reasonable performance in your opinion? It might not align with others, most people don't push that crazy amounts of data. I know IBM went from in-kernel NFS to Ganesha for their Spectrum Scale product recently.

[ilyt]

Ceph/cephfs have kernel clients (and FUSE ones too), but not server. Server is userspace.

It's easier to limit client attack space because to just start attacking client you'd need to MITM the client-server traffic

[rosnd]

"Crazy amounts of data" isn't the main concern, it's latency. It's the people storing giant amounts of data who generally don't worry about that so much.

[lillecarl]

We usually run those services with local nvme disks, they're not as portable but we get great performance.

[paulmd]

Ceph isn't a filesystem, it's a service layer (self-described "storage platform") that runs on top of some other unspecified filesystem. Think git-annex or hadoop, not ext4.

Anyway the way Ceph does that is replication, just like those other solutions. There may be 4 nodes with filesystems that contain that data, and Ceph is the veneer that lets you not have to worry about the implementation-detail of where it lives.

[Nullabillity]

Ceph actually does manage its own backing filesystem too these days, after the bluestore migration a few years age.