[DennisP]

Would Musk's argument about sensor fusion have made any sense back when they were doing lots of hand-coded C++?

I've been thinking maybe vision-only was a reasonable decision, back when lidar was expensive and the software was hand-coded. Now it doesn't, because lidar is cheaper and the software is and end-to-end neural net, and additional sensors are just more inputs to the network which will learn to use them. But Tesla is locked in because of the promises they made to early FSD buyers.

[AlotOfReading]

Sensor fusion is pretty straightforward. You can think of it like sorting algorithms in CS. There's a bunch of standard techniques simple enough to teach undergrads that work fine in production, and enough technical depth beyond them to last the rest of your career.

If you actually look back at the E2E tweet, musk only says that the NN replaced 300k lines of "control code". Control code usually doesn't encompass the entire AV software stack, but neither should it take 300k LOC. As far as I'm aware no one is 100% sure what they mean by E2E and if it's actually the standard meaning or something else that's been widely misinterpreted.

[maxlin]

Their engineers, who obviously are on the bleeding edge, going out of their way to avoid sensor fusion issues says something quite different. I could believe "Straightforward" could maybe apply for something many tiers below in complexity and safety requirements to what they're doing. But adding non-agreeing, non-uniform information sources to the most capable real-world ML vision system not driven by human-engineered code?

I don't care even if you said you had 70% of the experience their team has, what you say can't sound reasonable or caring for actually improving safety in numbers.

[AlotOfReading]

I've been through 3 public AV launches. I don't lead with that because "resume" measuring contests are boring and what I write should be evaluated on its own merit, not by who says it. This account is readily identifiable to anyone who knows me.

With that out of the way, it's much easier to write a hardware safety case than one for software. It's easier to write a software safety case for a traditional software architecture than one based on ML. It's much easier to write a safety case for focused models than an E2E system. None of these should be controversial statements.

You're arguing that Tesla is deliberately jumping to the hardest safety case in order to avoid the simpler safety case. I know many people who have worked on the relevant teams at Tesla. I don't think they're ignorant of the difficulty of Tesla's choices or making decisions based on what's easier to validate.

Sensor fusion is not the difficult part. You only have to look at the fact that virtually every other company in the industry (even the ones using E2E and not using LIDAR like Wayve) does it. Either we have to accept that everyone else is stupid, or there are other factors involved in Tesla's decision-making.

For what it's worth, I'm not sure if I could write a safety case about Tesla's system that meets my own personal standards. It's pretty clear from their actions and the various regulatory/legal inquiries that they don't have what I consider an effective safety process regardless.

[sidibe]

They go out of their way to avoid sensor fusions because they are not allowed to use other sensors. Some people seem to have no limits to their credulity.

[maxlin]

Yes. They are disallowing themselves from using other sensors. To avoid complicating the software stack and ultimately to keep it higher performing, which will make it safer in any alotted time.

This is really no different from good code practices with a complex system. Most HN readers should be familiar with rot in codebases quite comparable to "adding a few extra features to make it better" which just became a maintenance burden and take away from core features.

If you knew the slightest of how Musk has actually stayed exactly the same since the beginnings of Tesla, you'd know his hard specs are always technology-based. People that know more have said that his capability of speccing systems relatively right deep in to the future is possibly his single greatest leadership feat.

>Once Musk is near-certain about one technology pathway over another, he’s not afraid to put massive amounts of resources into that path, while still staying flexible enough in the case that a new emerging technology disrupts that particular path. Because he’s willing to make enormous (and seemingly risky) bets on these pathways, he’s able to outpace his competitors. https://www.quora.com/Is-Elon-Musk-all-that-hes-cracked-up-t...

[DennisP]

Meanwhile, pretty much everybody else doing safety-critical real-time control prefers multiple sensor types.

For an example of that in self-driving, Huawei uses vision, lidar, radar, and ultrasound. Out of Spec let it drive them around for an hour in busy traffic in a city in China. It looked about as good as FSD, without having had several million cars providing training data for years.

[queenkjuul]

People really act like SpaceX didn't rain fifteen tons of concrete onto employee's cars because musk decided he wanted to be the first person in history to launch a heavy rocket without a flame diverter, and none of the brilliant engineers were able to convince him otherwise or had the guts to otherwise scuttle the launch for the sake of safety.

Clearly Tesla engineers will do the same

[general1726]

> and the software is and end-to-end neural net,

Well considering Musk record for adherence to reality, question is if you can really believe that or if Musk thinks that this is happening or it is not happening at all and Musk is just making it up.

[DennisP]

Given what's happening in the rest of the AI field, it seems pretty likely that it's correct.

Plus various Tesla engineers have said the same thing, Tesla does have a very large AI training cluster, and FSD quality made a big jump when they claimed to deploy end-to-end.

[Jyaif]

You've got it exactly backwards.

When the software was hand-coded, having Lidar and a high-res map was vital.

But if you have a good enough AI, sensors that replicate the human senses are all what is needed.

The real question is: when will we get good enough AI that can be applied to all cars?

[DennisP]

The question isn't whether it can be done eventually, but whether it would get safer faster with extra sensors.

Back in the day, Elon's specific objection to lidar was that it was too hard to code the sensor fusion part. With end-to-end there's no coding to deal with.

And it's not like they've replicated the visual cortex. It's the same neural net technology everybody else is using. It can deal with any sort of sensors just fine.

[CyberDildonics]

> But if you have a good enough AI, sensors that replicate the human senses are all what is needed.

What is your example or evidence for this?

[Jyaif]

Concentrated and well-rested drivers with good eyesight not under the influence are able to drive very well.

[CyberDildonics]

So human senses are all we need and we just have to solve artificial general intelligence and fully simulate a human brain? I wonder why no one has done that yet.

[DennisP]

Yes, but we're a long way from putting AI hardware in cars that's as powerful as the human brain.

Plus our cameras aren't as good as human eyes.

[dham]

It's not "Musk's argument," it's Andrej Karpathy's.

Also, if you've ever done any ML you would note that more data isn't always better. Plus there's the piece about which thing to believe when you get conflicting data. It's a lot more to it than what random Hacker News people are saying in this thread.

[DennisP]

Conflicting data is an issue even when you just have lidar, and it's not hard to deal with.

The cofounder of Waymo taught one of the first Udacity courses on this subject. He went through a small Python project that processed lidar point clouds for self-driving. The data is noisy, you get conflicting information from different points, and the code aggregates all that into the most likely 3D model of the world.

Additional sensor inputs are just more of the same, and neural nets are pretty good at this sort of thing. They'd even learn which sensors are more reliable in different scenarios.

As for "more data isn't always better," I've mostly seen that applied to training, not inference in real-time control systems. Even for training, it turned out people had been fooled by a local maximum, and once past that, more data really was better.