Aside from challenges regarding per inference latency, any other unique challenges you guys faced when deploying nlp models to web? It's pretty cool to see ml being applied more actively in day-to-day web browsing.
One thing that we're trying to hit is the actual quality of the generated questions/answers; there's an immense amount of variance in internet content at large and it's difficult to ensure high-quality content without some kind of supervised QA (aside from some kind of hyperparam grid search). We were able to achieve some control over quality through rule-based heuristic filters on our generated questions, but we're trying to make this more robust.