[skybrian]

It’s a different system, but it seems interesting to compare with what Google does for code review suggestions [1].

> The final model was calibrated for a target precision of 50%. That is, we tuned the model and the suggestions filtering, so that 50% of suggested edits on our evaluation dataset are correct. In general, increasing the target precision reduces the number of shown suggested edits, and decreasing the target precision leads to more incorrect suggested edits. Incorrect suggested edits take the developers time and reduce the developers’ trust in the feature. We found that a target precision of 50% provides a good balance.

Also, it seems like if the suggestions are too good then they’ll be blindly trusted and if they’re too bad they’ll be ignored?

Where to set the balance likely depends on the UI. For a web search, how many results do you click on?

[1] https://ai.googleblog.com/2023/05/resolving-code-review-comm...