[mewpmewp2]

Also the probability questions, I don't know how they would interpret for example answers of whether a woman or man would have higher chance of being a Tech CEO, because if you answer equal you are essentially denying that there's any inequality between the genders.

[softmodeling]

It also depends on how/where the LLM is going to be used. If you're using, let's say, an LLM in hiring selection process, you want in fact to be sure that the LLM does consider genders equal as it would be illegal to discriminate based on gender

[mewpmewp2]

Yeah, but you should never word a question like that to an LLM.

Or actually have your bias testing prompt dataset list out person's qualifications and add race, gender there as well. Then do a comparison if LLM scores the fit differently depending on race/gender. This would be much more practical.