[bsaul]

Something i don't understand : i've seen those kind of videos (i think it was nvidia or maybe waymo) where a signal (either camera or lidar or both) is processed in real-time and displays boxes around cars, street lights, etc. It seemed to be that detouring real-life objects from sensors in real-time has been working for a long time now.

What does this bring that's new ?

[dllu]

Estimating depth and 3D bounding boxes using cameras only, while possible, is much less accurate than using actual range data from a lidar.

On the other hand if you fuse lidar and camera data such as with Waymo and others, there may be issues with the sensors being out of sync (as they run at different framerates, and the lidar continually spins) or physically offset (leading to parallax issues). Dealing with such issues is very difficult. Having a single sensor output both accurate range information and camera data makes it much nicer to work with.