[gouthamve]

The only one I know with "non-experimental" local-storage is VictoriaMetrics. But the big thing there is that data in VM is not replicated, so when you lose a disk/node, you lose that data.

Having said that, both Thanos and Cortex have experimental local-storage modes that are pretty good. You could also try them for now while they get production ready.

[simonrobb]

M3 provides local storage but is not experimental, on top of that with cluster replication which VictoriaMetrics does not provide, and has a kubernetes operator to help scale out a cluster.

Disclosure: I work on the TSDB underlying M3 (M3DB) at Uber. Still worth checking out though!

[Florin_Andrei]

> data in VM is not replicated, so when you lose a disk/node, you lose that data

The vmstorage component in VictoriaMetrics Server - is it RAID0-like (stripping) or RAID1-like (mirroring)?

https://github.com/VictoriaMetrics/VictoriaMetrics/tree/clus...

[valyala]

It is easy to implement RAID1-like replication in VictoriaMetrics: just set up independent VictoriaMetrics instances (single-node or clusters) and replicate all the incoming data simultaneously to these instances. This can be done either via providing multiple `remote_write->url` values in Prometheus configs or via providing multiple `-remoteWrite.url` command-line flags in vmagent [1]. Then query multiple VictoriaMetrics replicas via Promxy [2].

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/blob/mast...

[2] https://github.com/jacksontj/promxy

[valyala]

It is more like RAID0. VictoriaMetrics shards time series among available vmstorage nodes. I.e. each vmstorage node contains a part of data stored in the cluster. This is usually named shared nothing architecture [1].

As for data replication, VictoriaMetrics offloads this task to the underlying storage, since the replication is hard to make properly [2]. Proper replication must be able to perform the following tasks additionally to copying the data to multiple nodes:

* To heal the data (aka to return back replication factor) after a node becomes permanently unavailable. The healing process mustn't degrade cluster performance and it must properly handle other cases mentioned below.

* To gracefully handle temporary unavailability of nodes.

* To survive network partitioning when nodes are temporarily split into multiple isolated subnetworks.

* To handle data corruption.

* To continue accepting new data at normal rate when a part of nodes are unavailable.

* To continue serving incoming requests with acceptable latency when a part of nodes are unavailable.

* To replicate data among multiple availability zones (AZ), so the cluster should continue accepting new data and serving requests if a single AZ becomes unavailable.

I'm unsure whether popular systems that claim replication support can handle all the cases mentioned above. The only system that seems to handle these cases properly is GCP persistent disks based on Colossus storage [3]. That's why it is recommended storing VictoriaMetrics data on GCP persistent disks.

[1] https://en.wikipedia.org/wiki/Shared-nothing_architecture

[2] https://github.com/VictoriaMetrics/VictoriaMetrics/tree/clus...

[3] https://medium.com/google-cloud/persistent-disks-and-replica...