Hacker News new | ask | show | jobs
by base698 1589 days ago
Yes, and sometimes even listing can take days.

I worked somewhere that a person decided using Twitter Firehose was a good idea for S3. Keyed by tweet per file.

Ended up figuring out a way to get them in batches and condense. Ended up costing about $800 per hour to fix coupled with lifecycle changes they mentioned.

2 comments

> Yes, and sometimes even listing can take days.

You have a versioned bucket with a lot of delete markers in it. Make sure you've got a lifecycle policy to clean them up.

Doing an S3 object inventory can be a lifesaver here!