[AznHisoka]

Ruby and Sidekiq as the messaging queue

Postgres to store the data

Elasticsearch as a search index.

My ES cluster has around 10 nodes, 64 GB RAM, quad-core.

Postgres database cluster is 4 nodes, 1 TB, 64 GB RAM, quad-core.

800 crawler threads distributed across 10 dedicated servers.

[outpan]

Thanks a lot! This sounds reasonable. Did you guys look into professional services for this?

[AznHisoka]

Nope. We have lots of custom needs.