[whimsicalism]

By "the field", you surely mean the academic field. In the industry, we run controlled experiments to validate all the time.

Recommender systems is one of the few areas in ML where almost all of the knowledge is contained in industry, not academia.

[lmc]

That was my thinking - anything of value is product-specific and behind closed doors. It's not my field, but something I see come up from time to time that seems weirdly over-represented in ML articles.

[samhw]

I work on these systems, and if anything my only complaint about the field is the propensity to solve every optimisation problem with ML. I have seen people solve textbook-grade linear, and even differentiable, optimisation problems.

And the reason it happens despite the 'invisible hand' etc is because it still works, it just happens to be horrendously inefficient. I think that's the main area of inefficiency in the industry: not in getting the job done, nor even arguably in accuracy - at least not severely - but in overcomplicating the solution[0] because we've formed a cargo cult around one particular method of optimisation, beyond all nuance.

[0] I mean 'overcomplicating' in absolute terms. Of course the very crux of my point is that, from the data scientist's perspective, it's not overcomplicated - it's less complicated than using e.g. ILP precisely because we have made libraries like TensorFlow so incredibly easy and tempting to use.

[whimsicalism]

Fwiw, my org heavily relies on LP solvers in conjunction with ML to solve these problems

[monkeybutton]

Academia thrives on open benchmark that researchers compete against each other to get the highest score on. How would you replicate that with a recommendation system where in industry you test recommendations by variants A/B/... and observe the winner? Your benchmark is no longer static and you rely on users' feedback for the recommendations each algorithm made.