[tomgs]

Cool! Two questions:

1. Where do you see this observability for LLM thing going? What's the end game? Is it like in traditional observability where all formats eventually will converge to one format (which OpenTelemetry is trying to be)? I feel it might be a little bit early to tell, tho

2. I noticed you do auto-detection of the framework used, like LLamaIndex et al. Except for annotations, is there a deeper connection to the LLM framework used? This is auto-instrumentation, so I assume you do most of the heavy lifting, but should users of this framework expect some cool hidden eggs when they look at their telemetry?

[tomerf2]

Thanks!

1. Huh, good question. Hopefully there will be convergence. We started discussing with other companies in this domain to support or even switch to OpenTelemetry.

2. Nothing specific, except for - as you mentioned - being able to see trace of a RAG pipeline automatically.

[tomgs]

A quick one :)

While we're on the topic - how does traceloop factor into all of this? What's the connection between the two? I assume the former is the LLM observability platform (Datadog for LLM?) and the latter is your own auto-instrumentation thingie to supplement it?

[nirga]

Yes, Traceloop is kind of a Sentry for LLMs

[ssijak]

Its priced way high, 500$ for 50k LLM calls? 50k is not much at all.

[nirga]

The open source is free for all ofc. Our platform provides capabilities for monitoring and detecting hallucinations hence cost more.