|
|
|
|
|
|
by ur-whale
1699 days ago
|
|
|
> It does not know anything. I would argue that it "knows" an awful lot, but it can't actually reason with it. However impressive GPT3 type models are, I am not particularly convinced that they're much more than glorified hashtables. If the hash table is large enough, it can produce lot of answers to a lot of questions, or approximately imitate a lot of stuff it's seen before. Whether it can actually combine "knowledge" it has stored in its weights into a pattern it's never seen before ... I'm not convinced. |
|
|
There is a 1-1 correspondence between data compression and generative models. GPT-2 is a highly effective loseless data compression tool: https://bellard.org/textsynth/sms.html
Always wondered why this insight is not taught as much, especially in the context of things like dimensionality reduction...