[lmickh]

Interesting test, but I find some of these benchmarks kind of miss the point. Even Grafana's.

The appeal of Thanos/Cortex/Mimir is the long term object storage. The value isn't that it is simpler or cheaper to run. The value is that I can compare data to months if not years ago. It can cost much more than the price of the instances to store good metrics over time even when the data is rolled up.

Scaling the read/write path separately has a lot of benefits as well too, but I would guess that doesn't come up often for most folks.

How much telemetry you can get in/out of your system over a day is important, but how much you can get in/out of it over years is overlooked.

[valyala]

Both VictoriaMetrics and Grafana Mimir perfectly fit for long-term storage for Prometheus data. The difference is in the used data storage types - VictoriaMetrics stores data to persistent disks (aka block storage), while Grafana Mimir stores data to S3-like object storage. Both storage types - block storage and object storage - can be used for long-term storage. They have the following differences in the context of major cloud providers (AWS, GCP, Azure):

- Object storage space usually costs 2x-8x less than block storage space.

- Object storage has up to 100x highest latency for data access than block storage (hundreds of milliseconds for object storage vs milliseconds for block storage).

- Block storage usually has much lower network-related error rate comparing to object storagr. For example, it is quite common practice to retry reading data from object storage on network errors, while block storage-based filesystems are much more reliable for this aspect in major cloud providers.

- Cloud providers tend to charge every read operation for object storage, while reading from block storage is free. This point is usually overlooked when estimating costs for block storage vs object storage.

Given these differences, block storage usually provides better performance than object storage. Block storage also can cost less than object storage when the stored data is read frequently.

VictoriaMetrics is optimized for HDD-based block storage, so there is no need to use more expensive SSD-based block storage in most cases. Additionally, VictoriaMetrics compresses production metrics 2x-10x better than Prometheus-like solutions, which store data to object storage (Thanos, Cortex, Grafana Mimir). This also reduces long-term storage costs.

On top of this, enterprise version of VictoriaMetrics can be configured to downsample historical data, so it will take less disk space [1].

[1] https://docs.victoriametrics.com/#downsampling

[PeterZaitsev]

To be fair, the benefits of "object store" it is scalable and bottomless while you have to play with EBS volume expansion etc. Some folks find managing fleet of EBS volumes not a big deal others find it problematic.

I think having "long term storage" on S3 compatible location is a way to go but you need ability to use local storage as cache to queries on recent data or just date range you're working with can be fast.

[valyala]

Agreed with this. That's why we at VictoriaMetrics are investigating a hybrid storage scheme - to store recently added data at block storage, while gradually moving older data from block storage to object storage in background. On the query side, the requested data should be transparently queried from both object storage and block storage.

[hagen1778]

> The value is that I can compare data to months if not years ago.

I'd say it is quite specific case. The most important data is recent data. Monitoring system should help you to identify current issues. Be reliable and performant, so you don't spend minutes waiting for response while your production is on fire.

Second in importance is data for last N days. The period when you analyze recent changes (updates, releases) or incidents. You want this data to be easy to get and pivot, changing queries ad-hoc and get results immediately. So root cause analysis won't take days of work.

Data older than month is rarely accessed. It is usually used for capacity planning, retrospective analysis - things which you do once in 3 months, or even once a year. Here, you can afford long, slow queries.

Both, VictoriaMetrics and Mimir, do a lot to provide fast access to the recent data: to get it stored and to get it ready for queries.

[BurritoAlPastor]

In my experience, the business use of retaining years-old metrics is overvalued, because changes in the system tend to prevent any kind of long-term apples-to-apples comparison. You’ll have changed your metrics engine, or your collector, or your tagging strategy, or your hosting strategy, or your containerization, or your deployments, or etc. etc. Even if you can find the like metrics from last year, you can’t trust that they meant the same thing then that they do now.

[liketochill]

In industrial settings where the process does not change but equipment is overhauled it can be quite valuable To compare year over year