[danpalmer]

This is a decision that companies can take, and some do (although I'd somewhat contest the google example, I've had good experiences before working here, and I'm biased of course).

I think the issue will come when companies start dropping customers that don't fit certain templates. They'll be accused of using racist or sexist AI, or something like that, and it'll be hard to defend against that because AI is a black box. With humans you can say "we have a policy against this", or "we'll retrain our employees", but that won't be possible for AI in the same way. Yes you can "retrain" an AI, but that retraining looks a lot less like a corporate education program, and a lot more like an IT change request, and surprise surprise, we're back to buying software systems.

[portaouflop]

> They'll be accused of using racist or sexist AI

I don’t see how that would be relevant from the companies perspective. Companies get accused of this and much worse all the time and it almost never has any real effect, happy to be corrected