Hacker News new | ask | show | jobs
by Eridrus 780 days ago
It would be great for this to actually explain what sorts of metrics are being computed here beyond what you get for free by instrumenting the requests library.

From looking at the screenshots, it looks like it can monitor number of tokens, which seems useful, but I'm not clear why that needed a whole big project.

I feel like the stuff you actually want to monitor in prod for ML that you don't get from infra monitoring are things that are not trivial to drop in because you want a sense for how well the ML components are working, which is generally pretty application specific. Having a general framework for that seems useful, but not really what we have here, at least for the moment.

Also, it just seems a bit weird for this to have it's own UI. Part of the point of OTEL is so that you can send all your metrics to one place. Not totally possible all the time and turning metrics into dashboards takes time, but the point of OTEL seems to be to separate these concerns.
1 comments
patcher99 780 days ago
Hey! On the metrics front, we do basic metrics like requests, tokens and cost for now as all of the usefull information about llm is included directly in the Span Attributes of the Trace.

So what a lot of users have told me is the want to track their usage(cost/tokens) and want to keep an eye on the user interactions like prompts and responses, We are looking to add more accuracy based metrics too which is very highly asked to us.

Also re on UI: UI is optional, You can use the sdk to directly send otel traces and meyrics to your preferred destination. https://docs.openlit.io/latest/connections/intro

link
arwineap 779 days ago
On the span is an interesting attribute to view, but had you considered using otel meters/metrics for that instead?

I think cardinality in span attributes can be a problem, and meters are better for aggregating and graphing

link
patcher99 779 days ago
We do Otel metrics for the main things needed in dashboarding 1. Requests (Counter) 2. Tokens (Counter) (Seperate metric for Prompt Tokens, Completion and Total Tokens) 3. Cost (Histogram)

I did attach a Grafana dashboard too (Works for Grafana Cloud, Ill get something for OSS this week) https://docs.openlit.io/latest/connections/grafanacloud

Since Otel doesn't yet support Synchronous gauge + Users wanted a trace of the LLM RAG application, We opted to Traces which is quite standard now in LLM Observability tool.

Lemme know if you had something more in mind, Love getting feedback from amazing folks like you!

link