[MountainArras]

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender data too in order to get better results when predicting the targeted diseases within these groups.

I think it's interesting to think about instead attaching generic information instead of group data, which would be blind to human bias and the messiness of our rough categorizations of subgroups.

[genocidicbunny]

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations of various ailments that may have variance due to gender or ethnicity. Basically, if most data is gathered from those who have the most access to medicine, there will be an inherent bias towards how various ailments present in those populations. So your base data set might be skewed from the very beginning.

(This is mostly just to offer some food for thought, I haven't read the article in full so I don't want to comment on it specifically.)

[QuarterReptile]

>women tended to be underrepresented in western medical literature.

Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

[stevenbedrick]

There is indeed a lot of evidence of this but you've got the direction backwards- it's not that women avoid studies, it's that for a long time studies specifically excluded women. Ditto for people of different races. This is why these days (well, as of today, at least) the NIH has a whole set of very well-established policies around inclusion in clinical trials that include sex, race, and age: https://grants.nih.gov/policy-and-compliance/policy-topics/i...

And this isn't for "DEI" reasons, it's literally because for decades there used to be drug trials that excluded women and as a result ended up releasing drugs that gave half the population weird side effects that didn't get caught during the trials, or just plain didn't work as well on one group or another in ways that were really hard to debug once the drug was on the market. That was legit bad science, and the medical research world has worked very hard over the last thirty years to do better. We are admittedly not there yet, but things are a lot better than they used to be.

For a really interesting take on the history of racial exclusion and bias in medicine, I recommend Uché Blackstock's recent book "Legacy: A Black Physician Reckons With Racism In Medicine" which gave a great overview.

Oh! And also everybody should read Abby Norman's "Ask Me About My Uterus," it gives a fabulous history of issues around women's health.

[marcuskane2]

Also, lots of medical studies have been done on drafted/conscripted soldiers which were all men. As well as lessons learned from treating injured and sick soldiers.

European medical studies had few non-white members because their populations had few such people until recent decades.

Lots of workplace accidents or exposures have led to medical knowledge, which are massively disproportionately male.

[lfmhd]

> It's more believable (still not very) that they disproportionately avoid studies.

Women are definitely strongly underrepresented in medical texts, and it's not typically by choice: https://www.aamc.org/news/why-we-know-so-little-about-women-...

A lot of "the consensus" in medical literature predates the inclusion of women in medical research, and even still there things are not tested on women (often because of ethical risks around fertility and birth defects).

[dragonwriter]

> It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to

“Medical attention” and “coverage in medical literature” aren't even remotely the same thing, so dismissing a claim about the first based on your anecdotal experience of the second is completely bonkers.

[genocidicbunny]

There's a few factors here:

1. We're talking about a span of 200 or so years. There is plenty of modern medicine that is still based on now century+ old knowledge.

2. The feedback loop. If you were learning medicine in the 1950's, you were probably learning from medical texts written in the 50 or so years before that, when it's not unreasonable to think women would have been less represented. Those same doctors from the 1950's would then have been teaching the next generation of doctors, and they carried those (intentional or not) biases forward. Of course there was new information, but you don't tend to have much time to explore novel medicine when you're in medical school or residency, so by the time you can integrate the new knowledge, some biases have already set in. Repeat for a few generations, and you tend to only get a dilution of those old ideas, not a wholesale replacement of them.

3. If you've been affected by such biases as a patient, you're less likely to trust and be willing to participate with medicine, once more reinforcing the feedback loop.

I don't have any specific numbers or studies for you, but you could probably find more than a few that attest to this phenomenon. I hate to go with 'trust me bro' here, but my knowledge on this topic largely comes from knowing people that are either studying or practicing medicine currently, so it's anecdotal, but the anecdotes are from those in the field currently.

[bryanrasmussen]

Your location seems to be in Cox, Virginia, not sure how widespread beyond that your experience is?

Of course lots of people have already noted that being represented in medical studies is not related to doctor's visits, but I would like to talk about the doctor's visits observation.

At any rate one thing that might cause you to think that Women are receiving lots of medical attention, based on your anecdotal evidence from visits to doctors' offices, there is one type of medical attention that of course is almost all women and that is the medical attention that revolves around pregnancy. That might skew your perception.

Furthermore if AI models and doctors have a tendency to miss disease among women it would seem to me to be reasonable to assume that women would be in the doctor's offices more often.

Example of why this is:

You go to your doctor, there is a man there, doctor says you have this rare disease you need to go to this specialist - you will not see that man in the doctor's office again dealing with his rare disease.

You go to your doctor, there is a woman there that has the same rare disease, the doctor says I think it will clear up, just relax you have some anxiety. That woman will probably be showing up to that doctor's office to deal with that disease multiple times, and you might end up seeing her.

on edit: there was another example of why women might be in doctor's offices more often then men that I forgot, women tend, even nowadays, to be the primary caregiver and errand runner for the family, sometimes if you have issues with children or your husband etc. has had an appointment, needs to drop a sample off, etc. it may be that the woman goes to the doctor's office and takes care of these errands around the medical needs of the rest of the family, and thus you might go to a doctor and see a couple women sitting around and wonder damn, why all these women always being sick, when the meeting isn't even about them.

[adamhartenz]

The history of ADHD research is a common referenced example

[autoexec]

Part of it is that women are less likely to join studies (especially risky ones that might impact their fertility or the health of their future children).

Part of it is that men are seen as disposable and it's more socially acceptable to exploit and experiment on men. It was also much easier to deal with men historically since once women got involved everything got a lot more complicated. This was especially true in the past where women were so infantilized that their husbands/fathers were put in charge of their medical care/choices. Those backwards attitudes had some strange consequences. On one hand women were seen as the property of men who could get their wives/daughters institutionalized or even lobotomized for not conforming, but at the same time women were also seen as delicate over-emotional creatures who had to be protected and whose modesty had to be preserved in ways that just weren't a consideration when men were involved. Basically for a large part of our history both men and women have been treated like crap by society and while things have improved in a lot of ways, our records and knowledge have been tainted by those old stupid biases and so we're stuck dealing with the fallout.

[kaitai]

Here is an academic medicine perspective: https://www.aamc.org/news/why-we-know-so-little-about-women-...

To give you some TL;DR from personal-ish experience, women have historically been excluded from medical trials because:

* why include them? people are people, right? * except when they're pregnant or could be pregnant -- a trial by definition has risks, and so "of course" one would want to exclude anyone who is or could get pregnant (it's the clinical trial version of "she's just going to get married and leave the job anyway") * and cyclical fluctuations in hormones are annoying.

The first one is wrong (tho is an oversight that many had for years, assuming for instance that heart attacks and autism would present with the same symptoms in all adult humans).

The second is an un-nuanced approach to risk. Pregnant ladies also need medical treatment for things, and it's pretty annoying to be pregnant and be told that you need to decide among unstudied treatments for some non-pregnancy-related problem.

The third is just a difficult fact of life. I know researchers studying elite performance in women athletes, for instance. At an elite level, it would be useful to understand if there are different effects of training (strength, speed, endurance) at different times in the menstrual cycle. To do this, you need to measure hormone levels in the blood to establish on a scientific basis where in the cycle a study participant is. Turns out there is significant heterogeneity in how this process works. So some scientists in the field are arguing that studies should only be conducted on women who are experiencing "normal menstrual cycles" which is defined by them as three continuous months of a cycle between 28-35 days. So to establish that then you've got to get these ladies in for three months before the study can even start, getting these hormone levels measured to establish that the cycle is "normal", before you can even start your intervention. (Ain't no one got $$ for that...) And that's before we bring in the fact that many women performing on an elite level in sport don't have a normal menstrual cycle. But from the sports side, they'd still like to know what training is most effective.... so that's a very current debate in the field. And I haven't even started on hormonal birth control! Birth control provides a base level of hormone circulating in the blood, but if it's from a pill it's varying on a daily basis, while if it's a patch or ring it's on a monthly basis (or longer). There's some question of whether that hormonal load from the birth control is then suppressing natural production of some hormones. And why does this matter? Because estrogen for instance has significant effects on cardiovascular health, being cardioprotective from puberty up to menopause. (Yeah, I didn't even get started on perimenopause or menopause.)

Fine, fine, it's just data analysis & logistics. If you get the ladies (only between 21-35) into the lab for blood samples frequently enough and measure at the same time of day every time to avoid daily effects and find a large enough group that you can dump all the ladies who don't fit some definition of normal & anyone who gets pregnant but still get the power for your study, it's all fine, right? You've just expanded medical research to incorporate, like, 10% more of the population....!

[naijaboiler]

I am just tired of skeptics asking innocently. Yes I wish i could take time to look for sources to educate people like you, but I don't. So take my word for it or not. But yes women's medical issue are disproportionately underrepresented, misrepresented and understudied.

[da_chicken]

It's pretty well understood that there's an unfortunate bias towards white men in their early 20s. This is a pervasive sampling problem across all human studies because most researchers have historically been at universities. So their pool of subjects has naturally been nearby college students.

Just as those are the people who have historically been doing that research, the people who they have studied have been drawn from the same population. Over and over we find that problems from the assumption that the young, white, male college student is a model of "normal" for all of humanity.

Honestly, it's such a pervasive finding in medicine, psychology, and sociology that I think it says more about your relative inexperience in those areas than anything else.

[searealist]

Women use far more medical care than men. Men's insurance premiums subsidize women's.

[genocidicbunny]

Has this been consistently true for the past 200 or so years? Many medical texts are pretty old.

And how much medical care they use does not necessarily correlate with how represented they are in the training data sets for AI.

[searealist]

The burden of proof is on you.

[genocidicbunny]

Proof that health insurance premiums for men have been consistently subsidizing women's health insurance premiums for the last 200 or so years? Perhaps the practical non-existence of health insurance until the latter half of the 20th's century? Pretty tough to subsidize something that doesn't exist.

You also offered no evidence for your assertion in the first place.

[searealist]

The ACA bans health insurance companies from charging men and women different rates for the same coverage. Before this, Women would have higher premiums because, on average, they use their coverage more. This is very easy to look up.

I can cite the ACA, but you can not cite anything that says AI training sets are biased against women.

[belorn]

It is very true that a lot of medical knowledge is gained empirically, and there is also an additional aspect to it. The history of Medical research is generally studied on the demographics where such testing is cultural acceptable, and where the gains of such research has been mostly sought, which is young men drafted into wars. The second common demographic are medical students, which historically was biased towards men but are today biased towards women.

So while access to medicine indeed one demographic, I would say that studies are more likely to target demographics which are convenient to test on.

[gonzobonzo]

> The history of Medical research is generally studied on the demographics where such testing is cultural acceptable, and where the gains of such research has been mostly sought, which is young men drafted into wars.

Though in this study, the AI models were also biased against people under the age of 40.

It is interesting that we're also seeing a lot of bias in the reporting and discussion of these results. The results tested three groups for bias, and found a bias in all three. Yet the headline only mentions the bias against two of the groups, and almost the entirety of the discussion here only talks about bias against two of the groups while ignoring the third group.

If I test a system for bias, select three different groups to test for, and all three have a bias against them, my first reaction would be "there's a good chance that it's also biased against many other groups, I should test for those as well." It wouldn't be to pretend that there's only bias against the only three groups I actually bothered checking for. It definitely wouldn't be two ignore one of those groups, and pretend that there's only a bias against the other two.

[genocidicbunny]

I think we're really talking about different aspects of the same issue. Everything you've described basically agrees with "those who have more access to medicine" because those are also the ones inherently more convenient to test/observe.

[klipt]

Like how the ones with the most access to medicine are mice, because they're convenient to experiment on.

[genocidicbunny]

And this is absolutely something one needs to consider when reading medical studies -- if they only use animal (usually mice) models, there's a decent chance the conclusions are not directly transferable to humans.

[multjoy]

The key takeaway from the article is that the race etc. of the subjects wasn't disclosed to the AI, yet it was able to predict it to 80% while the human experts managed 50% suggesting that there was something else encoded in the imagery that the AI was picking up on.

[mjevans]

The AI might just have a better subjective / analytical weight detection criteria. Humans are likely more willing to see what they (or not see what they don't) expect to see.

[dartos]

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here.

For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women.

The biases and mistakes of those who created the data set leak into the model.

Early image recognition models had some very… culturally insensitive classes baked in.

[darth_avocado]

I am confused. I’m not a doctor, but why would a model perform poorly at detecting diseases in X-rays in different genders and races unless the diseases present themselves differently in X-Rays for different races? Shouldn’t the model not have the race and gender information to begin with? Like a model trained on detecting lesions should perform equally well on ANY X-Ray unless lesions show up differently in different demographics.

[ironSkillet]

You and the article are both correct. The disease does present itself differently as a function of these other characteristics, so since the training dataset doesn't contain enough samples of these different presentations, it is unable to effectively diagnose.

[garfield_light]

> [...] unless lesions show up differently in different demographics.

Well, first the model looks at the entire X-ray and lesions probably do show differently. Maybe it's genetic/sex-based or it's due how lesions develop due environmental factors that are correlated to race or gender. Maybe there's a smaller segment of white people that has the same type of lesion and poor detection.

[dartos]

> Like a model trained on detecting lesions should perform equally well on ANY X-Ray unless lesions show up differently in different demographics.

This is not true in practice.

For a model to perform well looking at ANY X-ray, it would need examples of every kind of X-ray.

That includes along race, gender, amputee status, etc.

The point of classification models is to discover differentiating features.

We don’t know those features before hand, so we give the model as much relevant information as we can and have it discover those features.

There very well may be differences between black woman X-rays and other X-rays, we don’t know for sure.

We can’t have that assumption when building a dataset.

Even believing that there are no possible differences between X-rays of different races is a bias that would be reflected by the dataset.

[watwut]

For a start, women have different body shape and you can (unreliably) tell a woman and from a men from an X-ray. The model can be picking up on those signs as a side effect and end up less correct for demographic it was not trained for.

[atlantic]

If diseases manifest differently for different races and genders, the obvious solution is to train multiple LLMs, based on separate datasets for those different groups. Not to mutter darkly about bias and discrimination.

[prasadjoglekar]

Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).

[genocidicbunny]

But they do look at bones and near-bone tissues, which can still have variance based on ethnicity and gender. For a really brute-force example, just think about how we use the shape of the pelvis and some other bones to identify the gender of skeletal remains of a person. If you had a data set of pelvic xrays that only included males, your data set would imply that female pelvic bones are massively malformed despite being perfectly normal for that gender.

[CJefferson]

This is the whole point of the article. Did you read it? Does the whole thing fail your sniff test?

Their results seem solid, and clear, to me.

[kaitai]

Breast density affects the imaging you get from x-rays. It is well-known that denser breast tissue results in x-rays that are "whiter" (I'm talking about the image of the tissue, in white, on a black background, as x-rays are commonly read by radiologists). Denser breasts are associated with less effective screening for breast cancer via mammogram. A mammogram is a low-dose x-ray.

When using a chest x-ray to look for pulmonary edema, for instance, I would be unsurprised if breast tissue (of any quantity) and in particular denser breast tissue would make the diagnosis of pulmonary edema more difficult from the image alone.

Also, you seem to have conflated a few things in your second sentence. Deep in the article, they did have radiologists try to guess demographic attributes by looking at the x-ray images. They were pretty good at guessing female/male (unsurprising) and were not really able to guess age or race. So I'm super interested in how the AI model was able to be better at that than the human radiologists.

[dartos]

There can be differences which statistical models pick up which we humans don’t.

For example, a couple years ago there was a statistical model made which could fairly accurately predict (iirc >80%) the gender of a person based on a picture of their iris. At the time we didn’t know there was a visible iris difference between genders, but a statistical model found one.

That’s kind of the whole point of statistical classification models. Feed in a ton of data and the model will discover the differentiating features.

Put another way, If we knew all the possible differences between someone with cancer and without, we wouldn’t need statistical models at all, we could just automate the diagnosis.

We don’t know the indicators that we don’t know, so we don’t know if some possible indicators show up or don’t show up in a given group of people.

That is the danger of wholly relying on statistical models.

[guhwhut]

Cancer progresses differently depending on ethnicity and sex. As does treatment and likelihood of receiving treatment at early stages.

Black women experience worse outcomes and are diagnosed with more severe forms of breast cancer than white women.

Cancer is not just one disease. Its progression will vary depending on type. If the AI is trained on only some strains of cancer, eg those traditionally found in white women in early detection scenarios, it might not generalize to other cancer types.

So yes, to your genuine question, medical imaging of cancer can vary depending on ethnicity because different cancers can vary between genetic backgrounds. Ideally there would be sufficient training data across the populations, but there isn't because of historical race bias. (Among other reasons.)

[sc68cal]

What groups have the financial means to get chest x-rays, and what groups do not? What historical events could create the circumstances where different groups have different health outcomes?

[ineedaj0b]

you ain't gonna like the truth but there are differences between the races and during med school they try to say it ain't so but once you start seeing patients there's differences in musculature/skin, all sorts. and if you have a good attending they tactfully tell you and you go 'was it in a study?' and nope nobody wants to publish it. and no i'm talking just stuff like scabies or diabetes.

[QuarterReptile]

Yes.

[ruytlm]

It disappoints me how easily we are collectively falling for what effectively is "Oh, our model is biased, but the only way to fix it is that everyone needs to give us all their data, so that we can eliminate that bias. If you think the model shouldn't be biased, you're morally obligated to give us everything you have for free. Oh but then we'll charge you for the outputs."

How convenient.

It's increasingly looking like the AI business model is "rent extracting middleman", just like the Elseviers et al of the academic publishing world - wedging themselves into a position where they get to take everything for free, but charge others at every opportunity.

[ElevenLathe]

We have to invent more ways to pay rich people for being rich, and AI looks like a promising one.

[genocidicbunny]

Do you think there is a middle ground for a progressive 'detailization' of the data -- you form a model based on the minimal data set that allows you to draw useful conclusions, and refine that with additional data to where you're capturing the vast majority of the problem space with minimal bias?

[bko]

Apparently providing this messy rough categorization appeared to help in some cases. From the article:

> To force CheXzero to avoid shortcuts and therefore try to mitigate this bias, the team repeated the experiment but deliberately gave the race, sex, or age of patients to the model together with the images. The model’s rate of “missed” diagnoses decreased by half—but only for some conditions.

In the end though I think you're right and we're just at the phases of hand-coding attributes. The bitter lesson always prevails

https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

[thaumasiotes]

> Also important was the use [in Go] of learning by self play to learn a value function

I thought the self-play was the value function that made progress in Go. That is, it wasn't the case that we played through a lot of games and used that data to create a function that would assign a value to a Go board. Instead, the function to assign a value to a Go board would do some self-play on the board and assign value based on the outcome.

[loa_in_]

X-rays are ordered only after doctor decides it's recommended. If there's dismissal bias in the decision tree at that point, many ill chests are missing from training data.

[darkerside]

Do you mean genetic information?

[pelorat]

I think the model needs to be thought about human anatomy, not just fed a bunch of scans. It needs to understand what ribs and organs are.

[ericmcer]

I don't think LLMs can achieve "understanding" in that sense.

[nomel]

These aren't LLM. Most of the neat things in science, involving AI, aren't LLM. Next word prediction has extremely limited use with non-text data.

[thaumasiotes]

People seem to have started to use "LLM" to refer to any suite of software that includes an LLM somewhere within it; you can see them talking about LLM-generated art, for example.

[hnlmorg]

Was it ascii art? ;)

[thaumasiotes]

https://hamatti.org/posts/art-forgery-llms-and-why-it-feels-...

People will just believe whatever they hear.

[satvikpendem]

Computer vision models are not large language models; LLM does not mean generative AI or even AI in general, it stands for a specific initialism.