[nullify88]

One thing that's slightly off putting about OpenTelemetry is how resource attributes don't get included as prometheus labels for metrics, instead they are on an info metric which requires a join to enrich the metric you are interested in.

Luckily the prometheus exporters have a switch to enable this behaviour, but there's talk of removing this functionality because it breaks the spec. If you were to use the OpenTelemetry protocol in to something like Mimir, you don't have the option of enabling that behaviour unless you use prometheus remote write.

Our developers aren't a fan of that.

https://opentelemetry.io/docs/specs/otel/compatibility/prome...

[valyala]

FYI, VictoriaMetrics converts resource attributes to ordinary labels before storing metrics received via OoenTelemetry protocol - https://docs.victoriametrics.com/#sending-data-via-opentelem... . This simplifies filtering and grouping of such metrics during querying. For example, you need to write `my_metric{resource_name="foo"}` instead of `my_metric * on(resource_id) group_left() resource_info{resource_name="foo"}` when filtering by `resource_name`.

[nullify88]

Thanks, that's nice to know VM can accommodate that. A migration will be something we will have consider if Mimir and OpenTelemetry force us to use joins for all our queries.

They maybe trying to address label cardinality but their approach seems like they are throwing the baby out with the bath water. The developer experience suffers as a result because from a dev pov, resource attributes are added to the metric yet this relationship isn't transferred when translated to Prometheus metrics.

[ronyaurora]

If you are using the prometheus exporter, you can use the transform processor to get specific resource attributes into metric labels.

With the advantage that you get only the specific attributes you want, thus avoiding a cardinality explosion.

https://github.com/open-telemetry/opentelemetry-collector-co...

[nullify88]

We've migrated away from the prometheus exporter to the prometheus remote write exporter as I'd like a completely "push" based architecture. Ideally I would have liked to be completely otlp but can't for the reasons already explained. So I use promethus remote write instead in to Mimir.

The transform processor could be useful if they ever deprecate the resource_to_telemetry_conversion flag, but its still a pain point because it hinders a developers autonomy, and requires a whitelist of labels to be maintained on the collectors by another team.