Hacker News new | ask | show | jobs
by petemc_ 2309 days ago
Can I ask how you implemented such a queue?
2 comments

One way for example would be store all events you want to write to ElasticSearch to a Queue (like rabbit, Kafka, etc) then have a set of workers/importers (this can even be logstash with input from queue and output ES) that read the events from that queue and store them in ES.

This way your system becomes a sync, and your system above is not affected if ES becomes slow writing or goes down for a reason. Your events will remain stored in the queue waiting to be processed.

We had our own database as a service thing that used ES and MongoDB. All teams used that service. Basically you would define your schema using the Thrift API and the service provided CRUD operations AND aggregations. This was a Java shop.

Scaling writes just meant changing that service's implimentation. We wrote them to Mongo and synched them to ES. There are much better ways to do that, but ES would be overloaded w/ just 500 writes a second so it didn't matter.