[demarq]

Also looking into the AWS limits;

> SQS had very low concurrent rate limits that could not keep up with the throughput of thousands of workers across the pipeline.

I could not find this perhaps the author meant Lambda limits?

> services like S3 have quite low rate limits — there are hard limits, but also dynamic per-account/bucket quotas

You have virtually unlimited throughput with prefix partitions

[wilsonzlin]

I'm not sure what were the exact limits, but I definitely recall running into server errors with S3 and the OCI equivalent service — not technically 429s but enough to essentially limit throughput. SQS had 429s, I believe due to number of requests and not messages, but they only support batching at most 10.

I definitely wanted these to "just work" out of the box (and maybe I could've worked more with AWS/OCI given more time), as I wanted to focus on the actual search.

[demarq]

Those are reasonable expectations. I’m very impressed with how it all worked out.