[joshvm]

You need both visible cameras and LIDAR (and RADAR) currently.

Cameras give you rich intensity information: you can easily identify road markings, signs, etc provided the scene is well illuminated. Depth from stereo is OK, but struggles if the scene is featureless. Cameras perform about as well as your eyes in inclement weather. The idea that cameras don't work at night isn't entirely true, because your car should have headlights.

LIDAR is used for robust distance measurement. You can make spot measurements on pretty much any non-specular (shiny) surface. It's active so it works without external illumination (so you get 360 3D vision at night, not just where your lights point), it's accurate enough for driving (cm-level at 10s of metres) and by paralleling sensors you can get realtime performance. The Velodyne system uses 64 rx/tx pairs for ~1mpt/s. In practice you get around 20k LIDAR points per camera image because those 1 million points are spread over a hemisphere and your camera is imaging at 30 or 60 fps.

[andygates]

Tesla seem to think they'll be OK with radar and cameras, and not lidar - that's the current hardware revision.

[joshvm]

True, probably should be an and/or, but almost certainly you'll have a better time using both - RADAR's spatial resolution isn't nearly as good. Also probably because if they tried to put a LIDAR rig into a consumer car it'd push the price considerably. Unless they could magically get the price down (as Waymo claim) it would be a healthy fraction of the cost of the car. The Velodyne x64 unit cost about $75k not too long ago.

You can do weird tricks with RADAR though, like the two-car lookahead thing that Tesla has implemented. You can do that with LIDAR if you're looking at a mirror (you'll measure the distance to thing in the mirror), but typically multipath returns don't have a high enough SNR to be useful.

[marcosdumay]

When does one ever drive in a featureless place? Are you talking about an empty street covered in snow? Or do you have any other situation in mind?

If a car can self-drive anywhere, but will refuse to take control in an empty street covered in snow, I guess that's not a fatal flaw.

[joshvm]

Street surfaces are relatively featureless (uniform grey) at low resolution or quality and a lot of matching algorithms will fail on them. With modern high resolution cameras and algorithms like SGM (see Heiko Hirschmuller's work) you can do pretty well nowadays, but it's not a panacea.

It doesn't necessarily need to be absolutely featureless: more specifically stereo matching suffers when there is local texture that is not sufficiently unique along an epipolar line (normally we use rectified images so epipolar = along the image rows). For a concrete example, if I showed you (or a computer) a small patch of road (< 15x15 px square) and told you to find its location in another image, you would struggle because of the ambiguity. This happens all the time 'in the wild'. Cars are shiny, which means specular reflections everywhere; global illumination differences are fine, but local differences cause problems. Matching surfaces like glass is also hard. Someone else mentioned the sides of artic lorries.

LIDAR avoids a lot of potential confusion, but I'm not suggesting that it's a catch-all. It's time consuming to scan and the data are sparse. The best systems (should) fuse data from all the different sources to maximise confidence.

[raisedbyninjas]

A light colored semi trailer against a light colored sky has already been a fatal flaw.