[aportnoy]

This is a fallacy of interpretation (translating verbal representations of events into mental representations of events), not of probabilistic reasoning.

The crux I think is that when

A. bank teller

is juxtaposed with

B. bank teller and feminist,

we vaguely and falsely interpret A as "bank teller and not feminist", while the correct explicit interpretation is "bank teller and possibly feminist, but possibly not".

[stfwn]

Just to add something that I found interesting when my logic professor said it at the time: while that “bank teller and not feminist” interpretation is strictly incorrect in a theoretical world, IRL it’s useful for humans to just assume pieces of information. Most people are not particularly feminist, so it’s probably safe to assume someone - especially a bank teller, maybe - isn’t feminist if it’s not mentioned.

Rather than a fallacy of some kind, people may actually be so good at correct inferences that they are bad at leaving that intuition out of their reasoning process when thinking about a weird, outlying theoretical world.

[iguy]

It doesn't seem surprising at all that we fail to understand abstract questions like this. The experts only learned to answer them correctly after millennia worth of mathematical development!

To study how accurately Bayesian some animals are, I think you need to find ways to pose them questions which are relevant to them, and read off their inferences from their behaviour. This is obviously harder to do, and you have to worry a great deal about the animal optimising for something that differs from your first guess (e.g. it doesn't want maximum food on a good day, it wants not to starve on a bad day). I don't have references to hand but I think that when we can do this, the results are quite good.

[gowld]

That's still a fallacy, even if the mechanism behind is successful on average. Assuming a bank teller is non-feminist is rational only if the feminism matters, which it does not. Even if 0 bank tellers are feminist, it's still unhelpful (and harmful is even 1 bank teller is feminist) to assume that an unknown teller is feminist for the purpose of the question. Prejudices are often statistically more correct than incorrect (though socially problematic), but in some cases are still flat-out inorrect, as they are in this example. Too to much of a "good" thing is toxic.

[esrauch]

The Wikipedia article on the effect mentions that the experiment has been done to correct that by explicitly wording it better and it reduces the magnitude of the effect but it still exists.