[PunchyHamster]

from my experiences

rustfs have promise, supports a lot of features, even allows to bring your own secret/access keys (if you want to migrate without changing creds on clients) but it's very much still in-development; and they have already prepared for bait-and-switch in code ( https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens... )

Ceph is closest feature wise to actual S3 feature-set wise but it's a lot to setup. It pretty much wants few local servers, you can replicate to another site but each site on its own is pretty latency sensitive between storage servers. It also offers many other features aside, as S3 is just built on top of their object store that can be also used for VM storage or even FUSE-compatible FS

Garage is great but it is very much "just to store stuff", it lacks features on both S3 side (S3 have a bunch of advanced ACLs many of the alternatives don't support, and stuff for HTTP headers too) and management side (stuff like "allow access key to access only certain path on the bucket is impossible for example). Also the clustering feature is very WAN-aware, unlike ceph where you pretty much have to have all your storage servers in same rack if you want a single site to have replication.

[runiq]

> [rustfs] have already prepared for bait-and-switch in code

There's also a CLA with full copyright assignment, so yeah, I'd steer clear of that one: https://github.com/rustfs/rustfs/blob/main/CLA.md

[antongribok]

Not sure what you mean about Ceph wanting to be in a single rack.

I run Ceph at work. We have some clusters spanning 20 racks in a network fabric that has over 100 racks.

In a typical Leaf-Spine network architecture, you can easily have sub 100 microsecond network latency which would translate to sub millisecond Ceph latencies.

We have one site that is Leaf-Spine-SuperSpine, and the difference in network latency is barely measurable between machines in the same network pod and between different network pods.

[PunchyHamster]

I meant that you can't span single cluster over high latency link, unlike for example garagefs

[__turbobrew__]

Ceph has synchronous replication, writes have to be acked by all replicas before the client gets an ack. Fundamentally, the latency of ceph is at least the latency between the OSDs. This is a tradeoff ceph makes for strong consistency.

[PunchyHamster]

I know. We run it for near a decade now. I mentioned it because a lot of uses for minio are pretty small.

I had 2 servers at home running their builtin site replication and it was super easy setup that would take far more of both hardware and work to replicate to ceph so while ceph might be theoretically fitting feature list, realisticially it isn't an option.

[yencabulator]

The Ceph way of doing asynchronous replication would be to run separate clusters and ship incremental snapshots between them. I don't know if anyone's programmed the automation for that, but it's definitely doable. For S3 only, radosgw has it's own async replication thing.

https://ceph.io/en/news/blog/2025/stretch-cluuuuuuuuusters-p...

Disclaimer: ex-Ceph-developer.

[victormy]

I don't think this is a problem. The CLA is there to avoid future legal disputes. It prevents contributors from initiating IP lawsuits later on, which could cause significantly more trouble for the project.

[overfeed]

Hypothetically, isn't one of the "legal disputes" that's avoided is if the projects relicenses to a closed-source model without compensating contributors, the contributors can't sue because the copyright of the contributions no longer belongs to be them?

[PunchyHamster]

exactly, CLA is there so project can be closed at whim of company and take all contributors work

[plagiarist]

Contributors cannot initiate IP lawsuits if they've contributed under Apache 2.0. CLAs are to avoid legal disputes when changing the license to commercial closed source.