[janalsncm]

Observability is great but it doesn’t give granular enough insights into what is actually happening.

Given a black box you can do two things: watch the black box for a while to see what it does, or take it apart to see how it works.

Observability is the former. Useful in many cases, just not here.

If you want to know what LLMs are actually doing, you’ll need the latter. Looking at weight activations for example, although with billions of parameters that’s infeasible.