[_puk]

> All this is to get back to the point that people want to audit the LLM before running the function because of the unreliability, there is hesitance with good reason.

some people - I think it's quite clear from this thread that not everyone feels the need to.

I'm now thinking requesting the LLM also output its whole prompt to something like a Datadog trace function would be quite useful for review / traceability.

[verdverm]

Most LLM observability tools do this

I'm currently using LangFuse and exploring OpenLit because it integrates with Otel, which you should be able to forward to Datadog iirc their docs

[kakaly0403]

Checkout Langtrace. It’s also OTEL and integrates with datadog.

https://github.com/Scale3-Labs/langtrace