[korostelevm]

AWS does the optimizations over time based on access patterns for the data. Should have made that clearer in the article.

The problem becomes unusual burst load - usually from infrequent analytics jobs. The indexing cant respond fast enough.

[ebingdom]

Thanks for the clarification. But now I'm confused about the limits:

> 3,500 PUT/COPY/POST/DELETE requests per second per prefix

> 5,500 GET/HEAD requests per second per prefix

Most of those APIs don't even take a delimiter. So for these limits, does the prefix get inferred based on whatever delimiter you've used for previous list requests? What if you've used multiple delimiters in the past?

Basically what I'm trying to determine is whether these limits actually mean something concrete (that I can use for capacity planning etc.), or whether their behavior depends on heuristics that S3 uses under the hood.

I'm fine with S3 optimizing things under the hood based on access my patterns, but not if it means I can't reason about these limits as an outsider.

[Macha]

S3 does a lot of under the hood optimisation. e.g. Create a brand new bucket, leave it cold for a while, and start throwing 100 PUT requests a second at it. This is way less than the advertised 3500, but they'll have scaled the allocated resources down so much you'll get some TooManyRequests errors.

[acdha]

Those are what I would assume for performance when the system is stable. The concerns come from bursty behaviour — for example, if you put something new into production you might have a period of time while S3 is adjusting behind the scenes where you'll get transient errors from some operations before it stabilizes (these have almost always been resolved by retry in my experience). This is reportedly something your AWS TAM can help with if you know in advance that you're going to need to handle a ton of traffic and have an idea of what the prefix distribution will be like — apparently the S3 support team can optimize the partitioning for you in preparation.

[ec109685]

Delimiter isn’t used for writes, only list operations.

S3 simply looks at the common string prefixes in your object names and uses that to internally shard objects, so you can achieve a multiple of those request limits.

aaa122348

aaa484585

bbb484858

bbb474827

Would have same performance as:

aaa/122348

aaa/484585

bbb/484858

bbb/474827