That's always been the catch for me; its an always-on remotely controlled camera-mobile, of which you happen to sit in the drivers seat and pretend you own while it drives you around dangerously.
Simply being always-online and sending telemetry, doesnt preclude the ability to remotely intercede in the car's operations.
I think there's precedent with the OnStar system in other cars, and there has anecdotally been evidence of various systems being operable by some kind of admin terminal.
I think "summon mode" is actually a great demonstration of how you can remotely access and do some form of piloting the car without a driver. There's also the disabling of features remotely on resale, and probably other examples.
Into the infotainment unit. I guess it's a question of your definition of "remote control". In the context of a car, I'd think that more than being able to change the aircon or turn up the volume.