[arzelaascoli]

We also shared an article about how we run these indexing jobs at scale at deepset with kubernetes, SQS, s3 and KEDA.

TL;DR: Queue upload events via SQS, upload files to s3, scale consumers based on queue length with keda and use haystack to turn files into embeddings.

This also works for arbitrary pipelines with your models, custom nodes (python code snippeds) and is pretty efficient.

Part1 (application&architecture): https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipeli... Part2 (scaling): https://medium.com/@ArzelaAscoli/scaling-nlp-indexing-pipeli... Example code: https://github.com/ArzelaAscoIi/haystack-keda-indexing

We actually also stared with celery, but moved to SQS to improve the stability.