[TaylorAlexander]

The infotainment system must be capable of taking address input, and that same computer would be internet connected for a variety of reasons. Taking over the steering as the above poster suggested would likely not be a desirable hack anyway. More likely someone would want to leave the low level control systems intact and just change the desired destination. A solid hack would actually wait until you get to your destination, and then once you send the vehicle to park itself, it instead just goes to the attacker's desired location. Might be a while before you even realized something was up!

It was noted that the Xbox One will accept voice commands from a video the console itself is playing. What happens when someone makes a hyper-targeted Pandora ad that uses your car's voice control function to enter a new destination address? If you are paying attention you will likely notice this, but many people have suggested that at some point you can sleep in your car and wake up at your destination, so even that isn't guaranteed.

No doubt direct control of steering and brakes will be highly locked down, but as you point out that in no way eliminates the possibility for mischief.