[jl6]

> While the study did not use the exact software companies like Tesla use to power self-driving cars because they are confidential, the software systems used for the study are based on the same open-source AI those companies use, according to Zhang.

I was under the impression that commercial self-driving software was deeply proprietary and confidential, and there is no way to know that this study will generalize if run on state of the art detectors. Tesla and Cruise are name-checked in the article - how do we know this isn’t a problem they have worked extensively on and made great improvements to, relative to the open source components?

Feels like a case of outrage-for-clicks.

[jnovek]

> Feels like a case of outrage-for-clicks.

The BI article is definitely outrage for clicks. I wouldn't be surprised if the actual journal article was more measured in its conclusions and this is just typical bad science reporting.

[dee-bee]

Presumably these companies are free to provide their software for research. The onus is on them to demonstrate it works in the first place....

> how do we know this isn’t a problem they have worked extensively on and made great improvements to, relative to the open source components?

They are a private, for-profit entity with a strong incentive to mislead people about their products. I see no reason to assume they've addressed this issue.

[candiddevmike]

The point of the article, to me, is computer vision will never be enough. These are machines and need to be augmented with radar and other object detection methods.

[ben_w]

Radar etc. almost certainly make it easier, but for it to be "never" this would also have to be provably impossible for humans.

Which isn't too say it's not never, as I remember studies in my own childhood that said human drivers were also bad at recognising how far away children were, and I've never heard of human perception of skin colour being tested in this way so it might just turn out that melanin is unfortunately good camouflage against tarmac…

…but unless and until that suggestion turns out to be correct of all humans, I default to assuming we're an existence proof of the capability to do without, and that means I still wouldn't say "never" to sufficiently advanced AI doing at least as well.

[wongarsu]

I'm confident that a comprehensive study would show that on average humans are worse at detecting people of color against dark backgrounds (if we agree this is code for "people with highly pigmented skin" and not Asians or Latinos). There is just much less contrast to work with, and dark skin also makes facial features stand out less (which is an issue because faces are the thing humans can recognize best).

There is a discussion we could have whether we want to measure self-driving cars against an ideal perfect baseline or against the status quo. But of course the ideal case is much easier to define, and has fewer things that make some people uncomfortable.

[seanmcdirmid]

I’m harder to see on a rainy winter night if I wear a black jacket vs a bright orange one. I learned early in it wear bright orange on those nights given that I was almost clipped by human driven cars a few time. Clothing choices are important.

[tbrownaw]

> But of course the ideal case is much easier to define, and has fewer things that make some people uncomfortable.

As another bonus, it also provides an extra excuse to advocate against replacing humans with AIs.

[sokoloff]

It’s intended to replace human drivers who have not yet evolved radar. I agree that radar could make it easier and/or more reliable, but there’s a pretty strong argument that building a system to equal/exceed humans using vision alone is possible.

[micromacrofoot]

Humans haven’t evolved radar, but computer vision systems are also worse than human vision systems in many ways, so they need to compensate

[sokoloff]

They are currently worse in many ways. So, perhaps the computer vision systems need to continue to improve until such time as they deliver clearly superior results to the as-observed outcomes of human drivers.

I worked on the vision system for an autonomous vehicle program in 1991, using the processing power available then. Our team held several world records at the time for different categories of completely autonomous travel on public highways.

If you fit any kind of curve between what (relatively little) we could do then for ~$200K in equipment and what a production car with < $1K of BOM costs can do today, it's reasonable to predict that well within my lifetime that vision-only autonomous driving systems could be better than a human on typical roads (absent snow cover).

[micromacrofoot]

> better than a human on typical roads (absent snow cover).

the weather caveats feel like evergreen statements about self-driving and make me feel like it's further off than most people realize — I agree that the improvements have been impressive over our lifetimes, but like in most general tasks there's still an enormous gulf between biology and technology

we'll get there eventually, and much faster than biology did (ie, not millions of years)... but I wouldn't be surprised if full self-driving was another 20 years out

[sokoloff]

I've made the same "never, ever will we completely replace human drivers" predictions before, informed by my experience trying to do it 30+ years ago but also by dozens of winters' worth of driving on snow-covered roads in New England.

But there are huge potential gains even if self-driving is only usable in 99.8% of driving scenarios, provided there's adequate safeguards and sensible hand-overs to human drivers. (Not dumping an out-of-control, at-speed automobile into the human driver's lap with 50 milliseconds of notice.)

[mola]

The human eye is a better "camera" than anything we use today in self driving cars (just read about the eyes dynamic range). Not to mention that human hearing has unbelievable audiolocalization ability that we struggle to explain.

[AdamN]

Isn't radar and object detection standard for these types of systems?

[candiddevmike]

Tesla does not: https://www.tesla.com/support/transitioning-tesla-vision#:~:....

[mrkeen]

They certainly should be.

Tesla stopped selling cars with radar in 2021: https://www.carscoops.com/2023/05/tesla-is-disabling-radar-s...

Elon Musk Overruled Tesla Engineers Who Said Removing Radar Would Be Problematic: https://insideevs.com/news/658439/elon-musk-overruled-tesla-...

[foooorsyth]

> Feels like a case of outrage-for-clicks

Like 99% of these “AI discrimination” articles.

>human-detecting AI is developed in a western country with ~60% white population. Most of the training data is collected there

>the AI performed slightly worse in Uttar Pradesh, where the people and everything else in the background look different

>AI is prejudiced! Get outraged!

Every time.

[mola]

Weird, you articulated the exact point of bias in AI but the tone you used is dismissing. Yes obviously AI is not a moral agent and it isn't racist per se. But if it's input is biased and the test is biased then the application will be biased. That's a problem, if you go and deploy these models where their training data is lacking. Let's say, self driving car using the AI you described deployed in Uttar Pradesh is less safe because of bias.

What is wrong with this statement in your opinion?

[foooorsyth]

I'm dismissive of endless streams of unhelpful clickbait articles written by barely-tech-literate journalists aiming to spark racial outrage. I'm not dismissive of the threat of bias in AI, and I'm certainly not a fan of cavalier automotive companies running clearly-alpha autonomous software on public roads.

Still, I find most of these kinds of articles to be obnoxious and unhelpful in their shaming. Did you read the paper linked in the OP? It investigates the following datasets: CityPersons, EuroCityPersons, and BDD100k.

* CityPersons data is from mostly Germany (with all of it coming from central Europe)

* EuroCityPersons, as the name implies, is data from European cities

* BDD100k data is from NYC and Bay Area

So are we shocked with the outcomes here, or are they more or less obvious? If I trained a popular object detector with pedestrian image data solely collected in India, would you be surprised if it performed poorly outside of India? Would that warrant a racially-inflammatory article title?

And, as others have pointed out, these types of investigations make the implicit assumption that autonomous companies aren't working hard to reduce these biases in their own internal training pipelines. As reckless as some of those companies are, I promise you that they are not solely relying on Western data before deploying to non-Western streets. The investigators here may have simply been lazy/biased themselves (only investigating openly-available datasets from Western regions), and then projecting that laziness/bias onto AV companies.