[SpaceMartini]

I have heard too many horror stories about Ceph (and OpenStack) to be confident about that. I certainly don't think I can truly beat S3 on cost or performance at the terabyte scale for household data - and while larger scale would give on-prem savings there are also higher expectations (in terms of availability and performance) of a multi-perabyte storage array.

[secabeen]

Really depends on your scale. At the Terabyte to 100s of TB level, you can solve most storage problems at minimum cost with NAS or ZFS on commodity hardware.

Ceph/Object storage comes into its own at the multi-petabyte and higher levels, which is not very many groups or institutions.

[SpaceMartini]

Solving storage at the tens of TB scale with commodity hardware is fine to a point (I have a ZFS NAS at home) but has much more ongoing maintenance burden than S3 and you need at least 2 copies for it to be a remotely comparable solution in terms of durability.

Ultimately you just have to design for what is important to you; I don't want to spend time managing this stuff any more, so keep a local NAS for my partner to access and put the bulk of my "cold" data into 2 different cloud object storage providers. Note that neither of these is actually S3; for business use I would absolutely use AWS but for personal files I can manage with the reduced capabilities and lower prices others offer.