How do you train it for emergency situations (i.e. a car suddenly turning left in front of you)? I'd imagine it would be hard to get many of those in the training data set.
Easy, you drive in a simulator! (ie: backfeed/simulate LIDAR and camera data using a video game (GTA5 for example)).
Then try to simulate lots of near-crashes, reactions to traffic lights, signs, etc.
Hotz's code just reacts to inputs. Simulate the inputs and you can run any training case you want.
Maybe partner with the creator of the XXX simulator series, you could make some kind of "MMO" where each participants have to run errands the safest way possible while interacting with each other, and upload the training data. Some player could be randomly elected as "maverick" whose goal is to crash and cause accident, the other player would have to handle them.
And if driving in highway is a problem, why not use a test terrain complete with fog generator? With RC car representing pedestrian, other car, animal, ... feed the video first into an AR system and then give it to the neural net.