|
|
|
|
|
|
by pplonski86
1816 days ago
|
|
|
Congratulatuions on launch! Models monitoring is really important and there is huge lack for tools to do this. How would you notify users (model owner) about potential problems? Are you going to use email notifications? Does user need to manually set threshold when alarms should be triggered? What to do if there is a data drift? What is the most difficult to detect drift example that you see? |
|
|
For the moment the workflows you describe are built externally: - To set notifications you can send the output from Evidently to other tools like Grafana and then build a notification workflow around it. If you have a batch model, you can use some workflow manager (like Airflow, or simply a cron job) to schedule a monitoring job at every model run and then log the results or send an email report. - Thresholds are manual. We learnt that the model owners usually have to tune them anyways since the models are very different (a small deviation in one model is nothing, in another is a disaster). But we plan to add the ability to generate default thresholds as the tool grows.
We are working on native integrations and tutorials for MLflow and Grafana in the next couple of weeks.
When you detect data drift, there are usually 3 options: - Retrain the model if you can (if you can label the data, for example) - Limit the model application (for example, tune the classification threshold, or exclude certain segments) - Pause the model or use a fall-back strategy (e.g. human-in-the-loop decision making)
Drift detection is really non-trivial when you have a lot of data (it will often show "drift" just due to the volume). We know some users need a solution for this use case, and plan to add something here.
Another aspect is that you have to be aware of which features are important for the model to not get too many false alarms.