[aleksiy123]

are there any tools anyone knows to collect this kind of telemetry while using the tools instead of offline evals.

running evals seems like it may be a bit too expensive as a solo dev.