[dmrivers]

Cool idea, some feedback:

1. Consider using conformal prediction to calibrate the cutoff. Conformal prediction provides a distribution-free guarantee under exchangeability. This would let you turn your raw probe score into a threshold with a guaranteed bound on the rate of wrongly-kept on-device answers. Source: https://en.wikipedia.org/wiki/Conformal_prediction

2. The best indicators of confidence in ML come from multiple independent methods. What was the result if you combine the token entropy and verbal confidence reporting methods? Does this improve the result?

3. I noticed you didn't mention the assessment method of rerunning the model and judging whether outputs are consistent. How does that method compare in terms of AUROC?

[HenryNdubuaku]

1. Thanks for this!

2. We combine both on Cactus; rolling entropy method helps with per-token signal for early exit, while the probe runs on complete output.

3. Users hate this in production, drains battery on edge devices, but ultimately primarily works ok-ish on 7B+ models, we will share the results too.