[eep_social]

Any plans on the roadmap for aggregation (compression) and long term retention or would you instead ship data back out to a separate tool?

I’ve enjoyed having 3-5 years of prom metrics retention to look at seasonal traffic trends but we spent a fair bit of CPU aggregating raw metrics down to the right granularity for that kind of long term retention. My feeling is that the observability world is moving towards small, localized installations with real-time data and separate systems with cheaper, slower, but long term retention. Curious how you see it as you’re building a product in the space.

[brancz]

Yeah that’s basically how this system is built as well. We buffer a certain amount of data in the ingestion nodes and then offload it to object storage. In object storage it’s actually laid out in the parquet format so we could allow users to export/download this. Or like I said, the object storage cost is really the smallest piece in the system we might even just decide not to have any retention at all.