|
|
|
|
|
|
by evgenyp
3706 days ago
|
|
|
One key ability is to batch updates. We just implemented Kinesis (AWS service similar to Kafka) to reduce load on our Elasticsearch database (~50GB) when running hundreds of individual jobs. Individual tasks (implemented in Celery, actually running off Redis) push to a Kinesis stream which is then consumed in batches by a very simple processor. |
|
|