[demarq]

The title should be “10x engineer creates Google in their spare time”

But seriously what an amazing write up, plus animations, analysis etc etc. Bravo.

It was also ironic to see AWS failing quite a few use cases here. Stuff to think about.

[demarq]

Also looking into the AWS limits;

> SQS had very low concurrent rate limits that could not keep up with the throughput of thousands of workers across the pipeline.

I could not find this perhaps the author meant Lambda limits?

> services like S3 have quite low rate limits — there are hard limits, but also dynamic per-account/bucket quotas

You have virtually unlimited throughput with prefix partitions

[wilsonzlin]

I'm not sure what were the exact limits, but I definitely recall running into server errors with S3 and the OCI equivalent service — not technically 429s but enough to essentially limit throughput. SQS had 429s, I believe due to number of requests and not messages, but they only support batching at most 10.

I definitely wanted these to "just work" out of the box (and maybe I could've worked more with AWS/OCI given more time), as I wanted to focus on the actual search.

[demarq]

Those are reasonable expectations. I’m very impressed with how it all worked out.