[plafl]

Do you feel that recommendations given to you are perfect? That more or less should answer the question.

Evaluating recommendation systems is hard because you actually require a human in the loop. Even worse, giving the recommendations alters the human behavior. Then you need to think what metric are you going to use. For training you will most probably use a proxy metric that correlates. Maybe you want to optimize different metrics and they actually need to be balanced. Then there are lot of confounding variables: maybe a better UX will improve the metrics than a better algorithm, or a change of products.

It seems that with big enough data you can improve old models with deep learning but I think recommenders are very far from similar gains to other fields (NLP or CV for example). And most companies don't have that much data.

[stepbeek]

Sure, but we don’t need to be sat next to them. A large e-commerce company can A/B test new models to improve. The success metric is revenue.

For smaller customer bases this is a tricky problem, but I’d argue that automated recommendations don’t work at a small scale anyway so manual curation is king.

[kqr]

If you do it right, automated recommendations work also for smaller customer bases. There's a lot of redundancy in customer and product data that can be exploited to generalise the behaviour of groups of customers over groups of products.

Doing this automatically is a huge positive ROI thing over manual curation. In fact, humans are not even that good at coming up with good recommendations. Manual re-arrangement has almost universally been an anti-relevance feature in A/B tests I've looked at.

But of course, almost nobody does the automation right.

[plafl]

That's the good part: putting in place a simple system will beat manual recommendations. Sadly even moderately big businesses lack sometimes automated recommendations. There are several companies that offer the service but I don't know how many clients they get. Why the disinterest? No idea, I value good recommendations as a customer but maybe it's not that relevant in the big picture.