[softmodeling]

It also depends on how/where the LLM is going to be used. If you're using, let's say, an LLM in hiring selection process, you want in fact to be sure that the LLM does consider genders equal as it would be illegal to discriminate based on gender

[mewpmewp2]

Yeah, but you should never word a question like that to an LLM.

Or actually have your bias testing prompt dataset list out person's qualifications and add race, gender there as well. Then do a comparison if LLM scores the fit differently depending on race/gender. This would be much more practical.