[clolege]

I’m still confused by just how good its responses and writing style are. I understand that it was trained on a large data set, but I feel like some training samples must have been weighted more heavily than others.

Did the training data incorporate how popular (e.g. likes or upvotes) each sample was as a proxy for quality? Or can you achieve this performance just by looking at averages on a large enough data set?

[krackers]

I wasn't too surprised by the poems and song outputs since those don't seem much more novel than previous SOTA, but OPs example of chatGPT being able to effectively simulate itself is really mindblowing. It means that "being able to model yourself and recursively reason about yourself" isn't a valid claim to distinguish human consciousness from whatever this network is doing anymore.

See also https://en.wikipedia.org/wiki/Attention_schema_theory (unrelated to the notion of "attention" used in ML)

[uptheroots]

Makes me think of Godel, Escher, Bach. It's been a while since I've read it but I recall Hofstader making the argument that self-reference is the key element of consciousness.

[jameshart]

What I find weird is that it generates well structured, well commented code.

Where is this massive repository of well structured code with good clear variable names that it’s tapping into?

[int_19h]

Maybe it just read all the StackOverflow comments about using clear, expressive names as identifiers, and then unlike most humans, it actually took them seriously? ~

[clolege]

Right. It must be weighting repos with more stars more heavily in this case, right?

[visarga]

You could see it as factorised skill learning. In one place it learns how to do some task, in another place it learns how to write nice code, then does both at the same time. It learns all the possible code styles and purposefully uses a good style because it has been taught to choose so - with reinforcement learning from human preferences.

[Xelynega]

One of the parts of building these generative models is building a classifier for how "good" their output is, otherwise the algorithm has no way to compare potential outputs in a generation.

That's one of the issues with these models, we say they produce "good" output but really they're producing output that is "good" from one specific point of view that happens to be expressed in code and introduces a large bias into their outputs.

[astrange]

“Good” isn’t expressed in code here. GPT3 was trained on a very loose problem (next word prediction). InstructGPT/ChatGPT are trained on reinforcement learning from human raters.

If it was all a computer program it’d be acting like ELIZA.

[Xelynega]

"good" for gpt was expressed in the way they chose the dataset to include.

Just because generative text models in the past(like ELIZA) were bad doesn't mean that the algorithms we have now are much more than better versions of the same.

[EMM_386]

I've had it write me complete stories, poems, life stories, etc that obviously have no training data, because I made them up.

The results were beyond impressive. I don't know how it does it, but it does it really well.