[bitL]

> Modelling and tracking other cars is a bigger issue, including pedestrians.

Unscented Kalman filters and particle filters are used for that - they are pretty good if you sample environment with >10Hz rate. Most of what you said has been taken car of pretty well, those aren't really hot issues anymore. Also, the thinking these days is that humans (even mediocre) can do a good job driving using eyes only and more robust camera solutions based on deep learning are here. Their problem is inference speed as HW is not fast enough to allow FHD or 4K input feed to be processed realtime, so imagery has to be downscaled significantly, leading to errors. Similarly, tracking other cars only depends on how big supercomputer can you fit into your trunk.