Camera vision and LIDAR perfectly complement each other. Camera vision is no good detecting unknown/outlier obstacles quickly and accurately. LIDAR is great at detecting unknown obstacles quickly and accurately.
You can tune the camera obstacle detection to be hyper-sensitive, which results in phantom braking, causing Passengers to feel that the car is "unreliable" while it actually is safer.
Humans are better at braking the appropriate amount when they see something strange, dynamically tuning their sensitivity in a new situation.
You can lax the sensitivity, which will reduce false alarms, but will actually cause more crashes, deaths, and injuries. You don't want your customers to feel unsafe, so from a business perspective you will inevitably reduce the sensitivity.
> What are the main challenges in building software that relies solely on camera input?
Probably the main challenge is that it took nature about a billion years to get to human level visual perception and understanding of environment and nobody really knows how to duplicate it.
You can tune the camera obstacle detection to be hyper-sensitive, which results in phantom braking, causing Passengers to feel that the car is "unreliable" while it actually is safer. Humans are better at braking the appropriate amount when they see something strange, dynamically tuning their sensitivity in a new situation.
You can lax the sensitivity, which will reduce false alarms, but will actually cause more crashes, deaths, and injuries. You don't want your customers to feel unsafe, so from a business perspective you will inevitably reduce the sensitivity.