The problem is that there is an enormous, nearly unignorable incentive to work around it. So they will.
As the customer base becomes more and more corporate (which it will), they end up with disproportionately more customers whose experiences cannot be used to train the model to make it better for those customers.
Either way, corporate customers cannot leach off the training from consumers handing over their personal data forever; there aren't enough specialists in that training set to improve the models with no loss of corporate trust.
This is not a conspiracy theory. It's futurology, maybe, but pretty basic stuff at that.
At some point, where does the training advantage for specialist LLMs come from, if not progressively encroaching on customer data for the benefit of equivalent customers?
Trade secrecy is all anyone has left. It's not paranoid at all. You would hope that most serious companies have a tier of corporate knowledge protection that is somewhere between Coca-Cola/KFC herbs recipe secrecy and Stringer Bell's note-taking exhortation: "is you giving the LLM notes on our unique advantage?"
As the customer base becomes more and more corporate (which it will), they end up with disproportionately more customers whose experiences cannot be used to train the model to make it better for those customers.
Either way, corporate customers cannot leach off the training from consumers handing over their personal data forever; there aren't enough specialists in that training set to improve the models with no loss of corporate trust.
Betrayal of their trust is inevitable.