[ripperdoc]

Some very valid points in this article, but we need to remember that ChatGPT is still "just" a language model and not an intelligent entity, and it's trained on massive amounts of texts containing the average of human knowledge and opionion that it will repeat back in a probabilistic manner. The training data will include large amounts of "bad" data, either in the sense of being insensitive but also in the sense of being incorrect. It can be trained to reduce the likelihood of "bad" things appearing in output tokens but that means fighting against two very big challenges: that it's trained on messy data to begin with and that the space of possible things to say (both in prompt and output) is so vast.

I'd guess every percentage point less bad outputs costs more than the previous, because it entails testing the output on more or better trained humans, or it entails using humans to filter the training data better.

I think in the end, knowledge in the collective human sense means not that you are just smart, but that you can judge, filter and trust sources - and even then, you will easily find n+1 acclaimed experts disagreeing on something fundamental. So being right, either morally or factually, isn't a deterministic thing either. Of course, humans aren't deterministic, but we can reason about what to say and not to say depending on the context.

[deelly]

What bothers me a bit: if we can't control/manage ChatGPT which is still "just" a language model and not an intelligent entity, then how do we be able to control/teach next much more sophisticated models?