[ctovena]

Nice article !

Super curious about your plan for persistence and compression ?

With TSDB interning labels, do you expect any increase of size for the label part ?

And finally any specific reason for not having this under the Parca repo ? IMHO working across multiple repo in go can be a PITA.

[brancz]

Great questions! We're planning on persisting data in parquet format in object storage, potentially with additional metadata to help the query planner optimize queries. In terms of compression, parquet already supports various modern compression mechanisms (zstd, lz4) that we already support if the schema specifies it. Though we have thought about potentially allowing different compression schemes to be used in different situations, for now it's static and part of the schema.

While the pure storage for strings of labels might increase, since we got rid of the inverted index entirely, the saving of that is greater than what we're spending on potentially duplicate strings.

We intentionally put it on the Polar Signals GitHub org to distance it from the Parca project. While we initially developed it for Parca, we think the applications can be much wider so we wanted to emphasize that. I do agree it can be a pain to have it separate, but for now we think having it separate is worth it.