[nemothekid]

I think you are missing my point; the problem isn't the question. There's a tendency to view ChatGPT as a some sort of "general intelligence" when it's probably better understood as a very powerful markov chain.

When the data used to train this model is crawled from the internet where the relationships of IQ and Race and mostly limited to the more extremist part of the internet; when you bring up those questions extremist answers may get weighted higher because those types of conversations tend to be outright banned on other platforms.

Therefore the model is likely to "confidently" give you answers that are probably extremist in nature and be very confident about it; which is a problem given that there's a lot of bunk information out there which is what I believe Scott Alexander is getting to here.