[FridgeSeal]

For most of the stuff we’ve deployed, we’re not yet operating at a scale/level of interest where A/B rearing is worth it. Additionally, the purposes we’re using most of these models for don’t really necessitate A/B testing.

When we do need A/B testing, we’ll probably use something like Seldon. As for predictions/second, not very much at the moment: 1 per 30 seconds maybe? It’s not deployed into a Kubernetes cluster because of scaling requirements, it’s because that’s where all our other services greet deployed till, and it’s more beneficial (ops and cost wise) to also deploy into there than it is to bother with having a separate workflow for deploying to lambda’s or SageMaker.

[streetcat1]

So how do you know if a new version of a model is better than the existing serving version?

[FridgeSeal]

As currently the only person doing data science things for the team, I’ll test to make sure changes I make to model/feature engineering/etc result in a better model. We’re not constantly, constantly retraining our models, because our incoming data and behaves the same. We’ve had the same model in prod for 4 months now; we don’t have any pressing issues with its predictions, and looking through the logs of what the input was the the output, it’s still performing as expected, so we’ll probably leave it longer.

[streetcat1]

I see, so how do you measure the difference between the incoming data and your training data?

Also, it looks like you have a very low volume of predictions?