[ra7]

Why are you equating sensor fusion with solving the entirety of autonomous driving? That’s disingenuous. You understand sub problems can be solved, right? Sensor fusion is just combining different sensor inputs so you have high confidence in your object detection and you’re not “confused with disagreements” as Tesla likes to say. Self driving involves solving prediction and planning problems too, not just sensor fusion.

[fastball]

I'm not equating it with solving the entire problem, I am pushing back against the idea that autonomous driving can be sub-divided into completely discrete sub-problems. So I guess you could say no, I do not understand that sub-problems can be solved.

Object detection is inextricably linked with prediction which are both inextricably linked with planning.

[ra7]

Of course, they are linked. It doesn't mean you need to operate worldwide to prove sensor fusion is working. There is no scaling issue with it. Sensor fusion is already proven effective as companies using it are showing excellent safety records where they are operate. The Tesla complaints about "what do you do in case of disagreements?" don't hold good anymore. They just don't want to invest in get it working (but are now forced to as they re-introduce radar in their vehicles).

[fastball]

What evidence do you have that there is no scaling problem with sensor fusion?

As you point out, the companies that have "solved" it don't have scale.

[ra7]

Sensor fusion has nothing to do with scale. They are merely algorithms that combine multiple real time inputs into a single low level representation. This process remains the same whether you operate in San Francisco or worldwide. How they are interpreted is what differs from a Waymo to a Cruise to a Tesla. Waymo/Cruise get richer input they can feed into their downstream perception algorithms. Look at Waymo’s object detection benchmarks against Argoverse, KITTI and Waymo Open Dataset for their performance. They top most of them.

They don’t have scale yet for different reasons (adverse weather driving, long testing cycles, costly operations setup). “If they don’t have scale, that must mean sensor fusion doesn’t work” is quite a ridiculous argument.

If you think sensor fusion is not possible, then you must also believe Tesla’s attempts at re-integrating radar will not work and that they won’t achieve their stated goal of FSD.

[fastball]

I never said I don't think sensor fusion is possible. My only argument has always been "adding more sensors does not mean you will have a better self-driving car". Richer data does not mean better data. It does not mean more actionable data either. All it means is that you have more data points. I absolutely agree that this can be a good thing, and probably is in the case of self-driving.

But maybe it's not. Maybe the additional flops required for sensor fusion would be more useful if given over to the planning part of the problem. After all, you don't have infinite cycles. This is what I mean when I say it has not been proven at scale. The edge cases matter (arguably they are all that really matter), and "solutions with more sensors work better at object detection in artificial benchmarks" is not nearly as convincing to me as it seems to be to you. Again, object detection does not exist in a vacuum, and a solution that generally behaves better in a very constrained environment might actually behave worse when you throw all the edge cases at it and add on prediction and planning.

[ra7]

> Richer data does not mean better data. It does not mean more actionable data either.

How actionable you make it depends on how good your engineers are. Diverse input data is definitely more actionable than just RGB pixels from images.

> But maybe it's not. Maybe the additional flops required for sensor fusion would be more useful if given over to the planning part of the problem. After all, you don't have infinite cycles.

On the contrary, it reduces your compute requirements. For example, adding lidar means you are getting direct distance measurements instead of wasting compute cycles predicting distance using voxels. It's a much more effective use of compute.