[darkwater]

> If you want distributed tracing, the service mesh can't propagate traces into your application (so if service A calls service B which calls service C, you'll never see that end to end with a mesh of sidecars)

Why not? AFAIK traces are sent from the instrumented app to some tracing backend, and a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in that request. Why a sidecar/mesh would break this?

[colonelxc]

I think the point is that the service mesh can't do the work of propagation. It needs the client to grab the input header, and attach it to any outbound requests. From the perspective of the service mesh, the service is handling X requests, and Y requests are being sent outbound. It doesn't know how each outbound request maps to an input.

So now all of the sudden we do need a client library for each service in order to make sure the header is being propagated correctly.

[darkwater]

But tracing cannot be done anyway with a sidecar and no modification to the service code anyway. With a sidecar (or eBPF) you will get blackbox metrics for free (connections throughput, latency, errors etc) but tracing it needs to be done inside the code (even automatically by some third-party library/addon or instrumenting manually). I understand the point that, once you are there instrumenting for tracing, you can also instrument for metrics and not use a sidecar. But to be fair distributed tracing is something that's catching on only now and metrics give you already some kind of visibility that it's better to have that not to have. On top of that you can add tracing and improve the observability.

[sciurus]

You described the problem correctly.

I think it's a stretch to say that requires a client library, though. It should be straightforward to have whatever library you are already using for http requests pass those headers through.

https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...

[sciurus]

> a trace-id is carried over via an HTTP header from the entry point of the request until the last service that takes part in that request.

But it's not, unless you specifically code your services to do that. Which isn't hard, but just means plugging an unmodified service into a service mesh isn't enough.

[afrodc_]

This. Header trace propagation is a godsend.