|
|
|
|
|
by quodlibetor
368 days ago
|
|
I recently wrote a similar tool focused more on optimizing the case of exploring millions or billions of objects when you know a few aspects of the path: https://github.com/quodlibetor/s3glob It supports glob patterns like so, and will do smart filtering at every stage possible: */2025-0[45]-*/user*/*/object.txt I haven't done real benchmarks, but it's parallel enough to hit s3 parallel request limits/file system open file limits when downloading.* |
|