Hacker News new | ask | show | jobs
by jhoechtl 362 days ago
I am relatively new to the topic. In the sample code of the OP there is no logging right? It's metrics and traces but no logging.

How is logging in OTel?

2 comments

To me traces (or maybe more specifically spans) are essentially a structured log with a unique ID and a reference to a parent ID.

Very open to have someone explain why I'm wrong or why they should be handled separately.

Traces have a very specific data model, and corresponding limitations, which don't really accommodate log events/messages of arbitrary size. The access model for traces is also fundamentally different vs. that of logs.
There are practical limitations mostly with backend analysis tools. OTel does not define a limit on how large a span is. It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.
> There are practical limitations mostly with backend analysis tools

Not just end-of-line analysis tools, but also initiating SDKs, and system agents, and intermediate middle-boxes -- really anything that needs to parse OTel.

Spec > SDK > Trace > Span limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-lim...

Spec > Common > Attribute limits: https://opentelemetry.io/docs/specs/otel/common/#attribute-l...

I know the spec says the default AttributeValueLengthLimit = infinity, but...

> It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.

...I'd love to learn about any OTel-compatible pipeline/system that supports attribute values of arbitrary size! because I've personally not seen anything that lets you get bigger than O(1MB).

Well yeah, there are practical limits imposed by the fact that these have to run on real systems. But in practice, you find that you're limited by your backend observability system because it was designed for a world of many events with narrow data, not fewer events with wider data (so-called "wide events").

OTel and the standard toolkit you get with it doesn't prevent you from doing wide events.

"Wide events" describe a structure/schema for incoming data on the "write path" to a system. That's fine. But that data always needs to be transformed, specialized, for use-case specific "read paths" offered by that same system, in order to be efficient. You can "do wide events" on ingest but you always need to transform them to specific (narrow? idk) events/metrics/summarizations/etc. for the read paths, that's the whole challenge of the space.
Logging in OTel is logging with your logging framework of choice. The SDK just requires you initialize the wrapper and it’ll then wrap your existing logging calls and correlate term with a trace/span in active context, if it exists. There is no separate logging API to learn. Logs are exported in a separate pipeline from traces and metrics.

Implementation for many languages are starting to mature, too.