A semi-off topic question. Is it not possible to get an accurate depth map based on a two camera stereoscopic setup? Like human eyes? Perhaps combine it with video processing to isolate objects at different depths.
I was watching a talk from Cruise that mentions this. The main problem with cameras is dynamic range. Dealing with different lighting conditions that can change quickly is hard (the sun is really good at washing out colors). Lidar doesn't care about the current lighting conditions.
Also heavy rain would be a problem for regular cameras. Not just seeing through the airborne droplets, but also (at a guess far more significantly) the water directly in contact with the windscreen causing severe random distortions.
- 3 different small cameras manually set at different settings, models chosen for their qualities handling light levels.
Those 4 live feeds were fed into a small black magic design quad layout device, that turned them into a single hd feed via hardware/real-time. That was fed into a hardware capture, that stacked the quad arrangement, applied some other filters and did hardware compression. At that point almost no latency was introduced but had a nice working base video feed. That was fed into the Linux box for processing.
The quad device created a sort of super hdr video, and the thermal layer took it to the next level. All of the cameras had drawbacks, but combined they were minimized.
the images need texture and edge detail to pick up on depth. Any portion of an image that is a solid colour would need to be interpolated in some way, possibly inaccurately. Occlusion is also a bit of a problem resulting in gaps in the depth map that need to be filled somehow.
here's an old comparison of algorithms. I imagine the state of the art has improved with Deep Neural Nets recently.
https://youtu.be/s-8cYj_eh8E?t=22m39s