| I would recommend using ClickHouse which provides very efficient compression and thus reduces the size of data drastically. Apart from that, it provides various other feature: - Dynamic datatype [0] which are very useful for semi-structured fields which generally logs contains very often. - You can configure column's & table's TTL [1] which provides efficient way to configure retention. At my previous job (Cloudflare), we migrated from Elasticsearch to ClickHouse and saved nearly 10x reduction in data size and got 5x perf improvement. You can read more about it [2] and watch the recording here [3] Recently, ClickHouse engineers published a wondering detailed blog about their logging pipeline [4] [0] https://clickhouse.com/docs/en/sql-reference/data-types/dyna... [1] https://clickhouse.com/docs/en/engines/table-engines/mergetr... [2] https://blog.cloudflare.com/log-analytics-using-clickhouse [3] https://vimeo.com/730379928 [4] https://clickhouse.com/blog/building-a-logging-platform-with... |