[Datenstrom]

We are framework agnostic for model development, models get converted to ONNX[1] and served with the ONNX runtime[2]. They are deployed as microservices with docker.

We are currently looking at MLflow[3] for the tracking server, it has some major pain points though. We use Tune[4] for hyperparameter search, and MLflow provides no way to delete artifacts from the parallel runs which will lead to massive amounts of wasted storage or dangerous external cleanup scripts. They have also been resisting requests for the feature in numerous issues. Not a good open source solution in the space.

Note that this is for an embedded deployment environment.

[1] https://github.com/onnx/onnx

[2] https://github.com/Microsoft/onnxruntime

[3] https://mlflow.org/

[4] https://ray.readthedocs.io/en/latest/tune.html

[hn2017]

Any issues with the relatively new ONNX format? How do you handle model monitoring, verifying accuracy of model over time?