[sourabh03agr]

Yes, general observability tools don't work well for ML applications as they lack support for ML specific use cases such as attaching Ground Truth label, data drift, model bias, etc.

Beyond that, requirements for Deep learning models is even more nuanced. Say, for language models, we provide two key features to effectively monitor them:

1. We represent the text by an embedding (e.g. BERT) which are much more informative from a statistical distribution perspective to find out edge cases, low density regions, etc. Further we use Earth Moving Distance to quantify data drift in the multi-dimensional space.

2. We allow user-defined smart signals to be written on top of your model inputs/outputs. You can classify a certain prediction to be wrong if it doesn't follow grammar rules or have occurrence of certain keywords or the prediction is followed by a certain user behaviour pattern (if an user is not satisfied with the response of ChatGPT, it is expected that they will ask the same question again and again in different ways). All these customisations serve as a good proxy to actually observe the model's performance and find avenues to improve them.