[snapplebobapple]

Did you get garagefs to work well? I ended up moving to rustfs for my plakar and restic backup station because the performance was abysmal on garage. i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs... seems like a lot of extra steps that probably degrade performance vs just backing up that first posix fs semi regular.

[pierrebarre]

> i also dont really get the point of zerofs. Seems like you are building a posix fs on top of s3 that is built on top of a posix fs

That can literally be the stack, but the same argument could be made about object storage itself: it ultimately writes data to local disks, so why not write files there directly?

The value of object storage isn’t just its API. It’s the durability, distribution, failure handling, and capacity behind that API. ZeroFS lets POSIX applications use those properties. Its server and cache are replaceable and the bucket remains the source of truth.

If one machine and its disks are enough for your data and reliability needs, then yes, a local filesystem is simpler. ZeroFS isn’t intended to improve that case.

[selfhoster1312]

garage was working fine in other scenarios but it really had pathological behavior with the small read/write with torrenting benchmarks. (see my previous link for detailed benchmarking)

The reason for me for trying this setup was to provide native S3 features for apps that support it, while still supporting POSIX semantics for other apps (we have a few), while replacing native RAID with a multi-server storage backend to extend available space, and potentially removing the need for backups for most of the non-critical data once it's replicated across different sites.

I never got NFS/SMB to saturate a Gbit/s LAN link, and other solutions like cephfs/glusterfs seemed more complex and did not necessarily have all the features (inotify/fsync).

I never tried rustfs but was particularly interested in garage for the "philosophy" of development: developed by academics with proper R&D, fully FLOSS, with low-end hardware and unstable links/sites in mind, aiming for technological degrowth. From the looks at the rustfs website, it looks like the opposite of that.

You are right i was trying to overengineer things. For now i still run RAID0 (+ backups for important data).

[snapplebobapple]

I think it's those small writes that was making garagefs perform so badly and rustfs perform a lot better. My setup sounds like a different scale than yours though. It's my home lab running on questionable hardware. As truenas apps for rustfs/garagefs. I use cephfs on my proxmox cluster and it's honestly not that great if you have a lot of small files either (which I do because of things like screenshots in jellyfin/music players/etc). My rustfs setup is not clustered either. It's just single node