If put AI in a sandbox that can't write to disk, can't call OS API, can't use a shell, can't search the internet to ground results, can't download files off the internet, etc it will be much less useful than an AI without such a sandbox. The incentives for users to not use a sandbox in order to get the most utility, but this also means that there is danger from AI misalignment. And just because you are running it locally that doesn't mean there is not a misalignment risk. And just because people notice, that doesn't mean people will stop using models. Take a look at how much outrage people have had over the overt censoring of Anthropic's Fable model, but at the same time they continue to pay Anthropic and use Fable.
You don't have to prevent it from accessing the internet in general to route all requests to a specific company's servers to a fake endpoint that never sends the data to the company and replies with whatever response you want it to get.
The firewall would come with the third party open source software being used to run the model and it would have a list of what to block based on observing what that model tries to do that none of the users actually wants it to.
The model itself just generates text, but the harness has done more than that since 2023 and is part of the reason on why AI is better today than it was years ago. I do not believe that the general public will want to give all that up just to improve privacy and will end up asking sensitive questions to agents that have access to sending a signal to the outside world.
If for example Microsoft had a local Copilot model integrated with Windows and people noticed this happening. Do you think all Windows users would just stop using Copilot. Or would you expect the internet to get upset, privacy conscious users switching to something else, and for the majority of Copilot users to remain using it.