[genderwhy]

> The goal is to be as unbiased as possible given that limited amount of effort.

So you are therefore biased. You have a finite set of resources, and you are choosing to allocate them in a particular way. That is bias.

You could equally choose to allocate those resources away from the majority, which would also be bias. Any time a human is making an editorial decision about how to allocate resources, you are introducing bias.

[wizeman]

> > The goal is to be as unbiased as possible given that limited amount of effort.

> So you are therefore biased

Yes, but significantly less than before. Which is the goal.

> You have a finite set of resources, and you are choosing to allocate them in a particular way. That is bias.

That "particular way" is to give more weight to opinions that are under-represented in your training data and give less weight to opinions that are over-represented in the training data.

This is called "removing bias".

> You could equally choose to allocate those resources away from the majority, which would also be bias. Any time a human is making an editorial decision about how to allocate resources, you are introducing bias.

So, in your view, bias can only increase, it can never decrease?

Even if that were so, you are admitting that not all data is equally biased. Which means that it is possible to feed less biased data to an AI.

And the goal is not for "a human" to make an editorial decision. It's for the opinions used for the training data to be representative of all people, weighed according to (a representative sample of) these people (so you wouldn't be giving more weigh to the opinions of one person versus another).

[genderwhy]

I object specifically to this:

> Saying "there's no way to avoid the bias" is just an excuse to get away with being biased

And specifically to the implied idea that there is an unbiased target that is achievable.

[wizeman]

I meant that if you say "there's no way to avoid the bias", it sounds like you're basically admitting defeat and not even trying to reduce bias.

I don't think we can achieve 0% bias, I agree with you on that. But I think that, if you decide to spend some amount of effort (let's call this amount "X"), it is possible to reduce bias compared to if you spent zero amount of effort.

And that furthermore, if you spend a "Y" amount of effort where X<Y, then you can reduce even more bias.

Obviously, at a certain point this would have diminishing returns, so presumably there is some sweet point where, even though you can't be 100% unbiased, you can at least say you made a reasonable effort to be unbiased and that your remaining sources of bias are unintentional (and probably, almost just as likely to go in one direction vs another).

To bring the conversation back to the original topic, I think ChatGPT/InstructGPT is actually being actively biased towards one political side as a side effect of RLHF being done by people from OpenAI, even if this bias is being introduced unintentionally.

It would be much better, for example, if OpenAI could somehow accomplish RLHF using a sample of its users as the AI trainers.

It would still be a far cry from reaching 0% bias asymptotically, but it would already be an improvement, I think, as I think its users are a more representative sample of the population than OpenAI employees.