[patall]

One thing that I could imagine: it bases it's decision only on few (or the 'wrong') features while you (or marketing) want to consider more.

We have had a project where we were asked if our model would consider X. So we added X to the model but this didn't increase performance. Now the sane, simple answer would be to just ignore X. But then people come and ask why, doubt that it doesn't improve results, competition without ML considers X.

That doesn't happen (or is hidden) in a none ML situation where some decisions aren't questioned by a benchmark.