[mschulkind]

While I agree with basically all of this, and find the FSD on my Tesla to be quite useful, a question pops into my mind.

Why can't Waymo ALSO develop the same smarts and just also solve the sensor fusion issue such that they can use the right set of sensors in the right environmental conditions, and then leapfrog Tesla's capabilities?

[briandw]

Because they don't have a fleet of millions of people labeling the data for them and paying for the privilege of doing so. Waymo has about 3700 vehicles. Tesla has millions. Waymo only operates in known environments and collects a very limited range of data. Tesla collects data everywhere that people drive their cars.

[ai-x]

I thought about this and I think it boils to how the model is trained.

Tesla trains it models from actual drivers purely based on (input) Vision and (output) actuators - Brake, Steering, Accelerators.

Human output is based on what they and the camera sees. So, it's a 1:1 match.

If Waymo were to do that, it'll muddle the training set. The Lidar input may override camera input.

I always struggled when Musk mentioned Lidar will make it ambiguous. It didn't make any sense to me why having a secondary failback sensor messes things. But, if you put it in the training data context, it absolutely makes sense.

[mschulkind]

This is an interesting viewpoint, but isn't it also solveable?

Just because the human in the scenario only took vision as input, why does that matter to the training data and the model? The actions are the same.

To put it another way, what about all the cultural context the human had, or the sounds, smells, past experiences at the same intersection, etc? Even Tesla can't record this, but I'm not sure that matters.

[ai-x]

E.g If the driver brakes because they saw a pothole, and Lidar captures someone biking 200m away on their own path, it may mistakenly put more weight on brake causation to the 200m away object (because large moving object) vs the pothole.

I'm exaggerating, but I hope you get the point. It isn't even conflicting sensor signals about the pothole, but conflicting information about the causation. With vision only there is no conflict for the training data. This was my Aha moment. Multiple Sensors are absolutely important for fallback and extra safety, but screws up training that are based on Human Drivers

I think Elon himself doesn't understand this and hence can't articulate it, while just repeating whatever his ML engineer has said.

[tintor]

The biggest issue with using both camera and lidar is how to properly resolve conflicting returns from different sensor types.

[KennyBlanken]

That is vastly preferable to slamming into the back of an emergency vehicle because the cameras are dazzled by the strobes, or slamming into tractor trailers because the cameras were blinded by sunlight. Or slamming on the brakes because the car thinks a shadow in the road is a physical object...

[plqbfbv]

> such that they can use the right set of sensors in the right environmental conditions

Because this part is really hard, and that's why Tesla abandoned the fusion approach. You cannot possibly foresee all the conditions in which LIDAR or any active sensor will malfunction/return wrong data/return data that's only slightly off for that ONE specific time. And even if it doesn't, you need to trust it to not return noise. And when it does return noise, how do you classify it as noise?

Cameras are passive sensors - they get whatever light comes in and turn it into an image. Camera is capturing shapes that make sense to the neural nets: it's working. See all black/white/red/cannot see any shapes? Camera is not working, exclude it from the currently used set of sensors or weigh it less when applying decisions, because it's returning no signal (and yes, neural nets have their own set of problems).

EDIT: cameras also provide more continuous context: if 1 pixel is off, is clearly bright red in a mostly-green scene where no poles can be identified, the neural net will average it out and discard it as noise. If 1 pixel says "object" in LIDAR, do you trust it to be correct? Perhaps the ray just hit a bird or a fly, but you only see a point, it's a lossy summary of the information you need.

[mschulkind]

But why can't you apply all that same logic and processing to LIDAR as well. Maybe we're not there yet, but about about in 5-10 years when we are?

[tintor]

There is noise on LIDAR returns too. No one considers a single LIDAR point to be a collision hazard.

[ACCount37]

They could in theory. If they put at least as much emphasis on the AI side as Tesla does. Or if someone else cracked vehicle AI wide open and left it open for them to copy, and then they did exactly that, and found a way to bolt on their extra sensors in a useful fashion while at it.

As is, Waymo's playing it smarter than Cruise did, but they're not all in on AI yet. So I don't expect them to "leapfrog Tesla" in that dimension - and it's the key dimension to self-driving.

[tintor]

The main reason Tesla's don't have LIDAR is hardware cost and maintenance cost, not improved safety.

[SoftTalker]

Maybe also that cars with a LIDAR rig on the roof are appallingly ugly.

Tesla wants to make EVs that look like normal cars (Cybertruck being the oddball here, admittedly).

[throwaway2037]

If I recall correctly, Elon chose to avoid LIDAR (in favour of cameras) due to cost, not cosmetics. Am I wrong?

[SoftTalker]

I don't know. Cost might have been part of it but I also recall hearing that he thought since humans can drive with two eyes and no LIDAR then the car should be able to do the same thing.

[tintor]

It is not about the look. There are small solid state lidars that Tesla could have used.

[CSMastermind]

I got downvoted for saying this last time the topic came up but constraints focus a project. It’s best to start work with as few variables as possible, and only add new ones when absolutely necessary.

I'm working on a similar problem in computer vision and we're quickly approaching the point where our pure vision work is better than our Lidar supported track because we've had to deal with the constraints instead of having a crutch to lean on.

[mschulkind]

I agree, but these are also the exact constraints that lead to an early leader getting overtaken by a longer term, yet better set of plans. Not saying that's the case here, but given how much success Waymo has had so far, over really everything Tesla has produced, says quite a bit about the likelihood of the approach, even if it's not yet there.