Hacker News new | ask | show | jobs
by ml_basics 1387 days ago
Interesting, thanks for sharing.

> I'd seen too many legitimate situations in which a metric declines even though there's no regression, such as bug fixes in metrics or occasional updates to the test data.

I guess you could keep both the old and new versions of the test data or metrics in parallel for a few cycles of model update to get a feel for how the old/new settings differ