Hacker News new | ask | show | jobs
by renezander030 2 days ago
The failure mode I would want addressed before putting this in front of an agent fleet is silent quality regression. When a turn gets routed to a cheaper model and the agent still completes the task, the trace reads as a success, and nobody attributes the worse output to the routing decision until it has compounded downstream. In the enterprise pipelines I build the spend win rarely came from per-turn model choice anyway, it came from cutting how many turns hit an LLM at all, with models pinned per workflow step so a regression was attributable to one change. That pinning is what made the system auditable, which is what a buyer's platform team actually signs off on. Do you run a shadow sample where routed turns are also executed on the frontier model, so a customer can see the measured quality delta instead of trusting the router's own confidence estimate?