[simonw]

Is the term "ChatGPT" being used in place of GPT-3 here? Is this thing actually replicating the GPT-3 training process?

The thing that makes ChatGPT interesting (over regular GPT-3) is the RLHF process, but this article doesn't seem to touch on that at all, unless I've missed something.

[de6u99er]

GPT-3 has been publicly covered in scientific publications. Same as GPT-2, and GPT. Those are all pre-trained models, where GPT is the abbreviation of Generative Pretrained Transformer. Transformers have been invented in 2017 at Google Brain [1].

-> https://medium.com/walmartglobaltech/the-journey-of-open-ai-...

GPT-4 is around the corner, and it's allegedly 100x more powerful than it'd predecessor.

-> https://medium.com/geekculture/gpt-4-100x-more-powerful-than...

[1] https://arxiv.org/abs/1706.03762

[wcoenen]

That source about GPT-4 is nonsense. It claims GPT-4 will have trillions of parameter, and at the same time links to another page which says that it won't be much bigger than GPT-3:

https://www.datacamp.com/blog/what-we-know-gpt4

[simonw]

That "100x" figure is extremely poorly sourced. I don't believe that at all.

[imtringued]

And yet the intimidating pictures of a small and large circle keep getting posted everywhere.

[de6u99er]

You're right. Apologies for that.

[rnosov]

Surprisingly, they are using the term correctly. Although it seems that the main point of the post was to plug their "Colossal AI" framework but if you do an in-page search for "Low-cost replication of ChatGPT" subheading midway in the article they do claim to replicate RLHF thingy fully whatever it might be. Interestingly, they also suggest that it would work with both BLOOM and OPT meaning that you can potentially make things like ChatBLOOM and ChatOPT (even on a consumer grade GPU). Lack of demo doesn't inspire too much confidence though.

[faizshah]

The article talks about their RLHF implementation briefly. There’s details on their RLHF implementation here: https://github.com/hpcaitech/ColossalAI/blob/a619a190df71ea3...