[AndyNemmity]

We are currently using Elasticsearch, and can only store a tiny amount of the data because of how much we use. 5 days at most.

ClickHouse is replacing Elasticsearch in this context, and is providing the same data storage, but with better compression, and not holding data in memory?

Is that correct?

[isburmistrov]

I think the storage architecture in ClickHouse and Elastic are very different. And I think compression in ClickHouse can be really damn good.

Don't have a good comparison at hands though, but a few random links on the topic:

https://engineeringat.axis.com/schema-changes-clickhouse/ https://clickhouse.com/blog/contentsquare-migration-from-ela... https://clickhouse.com/blog/100x-faster-graphql-hive-migrati... https://blog.zomato.com/building-a-cost-effective-logging-pl...

[AntonFriberg]

I am the author of the first link.

We successfully replaced Elasticsearch with ClickHouse within a Year. We went from having difficulties managing 3 months of data to storing 5 year+ of data. We also had difficulties onboarding users into Kibana and Elasticsearch world, this was not such a big problem with ClickHouse since most developers feel comfortable with SQL.

This change together with Apache Superset for the BI layer made a huge impact in the amount of internal users that could extract value from the data collected. Went from around 150 to 800 internal users.

We have not yet managed to bring everything into a wide event table but as long as the logs can be joined with the metrics under an interface that the developers feel comfortable with and the solution is cost-effective enough to allow all relevant context to be collected you will get far imo.

[isburmistrov]

Thanks for sharing!

> This change together with Apache Superset for the BI layer made a huge impact in the amount of internal users that could extract value from the data collected. Went from around 150 to 800 internal users.

Really impressive results.