[JamesSwift]

Well its not that simple. In the same way that throwing an LLM into a process will always have a risk of blowing up spectacularly.

In this case it failed open. It didnt recognize that it was in an edge case (which itself is an edge case). So what are you proposing to be the solution to that? If the car itself does not recognize that its in an abnormal situation that needs intervention then how do you intervene?

[Fernicia]

His point is that humans are prone to the same error. The flooded engine damage doesn't come from humans recognising the danger of a flooded road and choosing not to attempt it.

[JamesSwift]

Im responding to the implication that you "have to be ok with good enough" and that somehow this will be a mostly fine autonomous experience with this

> A car that only fails in a road conditions edge case is good enough for the vast majority of cases. You accept that, and issue a manual override for when that edge case pops up

But its just like LLMs. They will never be perfect, and so if you arent actively paying attention and steering the behavior then there is always a risk of spectacular failure. Because if you arent paying attention to "needing to [apply a] manual override" then all of a sudden the AI has `rm -rf /` and you had it in "bypass permissions" mode.

[sarchertech]

No one cares about flooded engines that Google has to pay for. They care about a taxi that might kill them.

You have to compare this to the number of taxi and Uber drivers who will drive into moving water with passengers on board while a passenger is telling them to stop.