[gregsfortytwo]

Just for future reference, RADOS is actually not very S3-like. It is an object store; it does map from object names to buckets of bytes. But unlike S3 and many similar object stores or key-value DBs, RADOS allows you to do file-like operations: you can append, write to random offsets in the object, overwrite pieces of it but not the whole object, etc. (That's all in addition to some stunningly-complex stuff like injecting custom code to do specific kinds of transactional read-writes on the OSD [storage node] itself.)

That's all key to RBD being useful, or indeed CephFS itself. There are systems that map a filesystem layer on top of S3, but they have trouble because there aren't good ways to overwrite random small pieces of an S3 object. With RADOS, there are! :)