| I have 2TB of photos across a 6-drive btrfs array. External drives from 2004,
NextCloud phone backups, scanned photos from the 80s. Some files
duplicated 3 times across forgotten folders. Finding duplicates was always a
weekend project that never happened. Then I realized: if an object store knows every file's content hash, duplicates
are just a GROUP BY. Built a tool that serves local files via S3 API: docker run -it -v ~/Photos:/data -p 9000:9000 ghcr.io/deepjoy/shoebox /data
Credentials auto-generated and printed on startup. Works with rclone, AWS CLI,
any S3 SDK. Files stay exactly where they are.The duplicate detection I wanted is one query. But the real surprise: every S3
tool just works. I set out to find duplicate photos and accidentally built a
local S3 server. What it is: S3-compatible object store for local filesystems. Rust, axum, SQLite
for metadata. MIT licensed. Or build from source with `cargo install`. What it isn't: not distributed, not for petabyte scale, not a MinIO replacement.
MinIO is built for production clusters. This is built for the NAS in your closet.
Single node, single process. If you need multi-machine replication, use MinIO or
SeaweedFS. Tested on real hardware this past week. A few early users ran it against their
own NAS setups on btrfs, ext4, and ZFS. Built a companion webapp to browse
duplicates visually (duplicate detection is non-standard S3, so terminal demos
weren't compelling). GitHub: https://github.com/deepjoy/shoebox
Companion webapp: https://deepjoy.github.io/shoebox-webapp/ |