[autophagian]

Do you have any resources you'd recommend to form a better understanding of how NNs tick? I'd like to get a better intuitive grasp on whats going on - I've mostly just been responding to that with "Well, if stochastic parrotism can do all this..."

[probably_wrong]

In case you haven't seen it yet, the term "stochastic parrot" got introduced by this paper [1] titled "On the danger of stochastic parrots". A related paper [2] titled "Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data" got awarded Best Paper by the Assoc. for Computational Linguistics and it's also easier to read.

Those two papers are critical of LLMs and discuss what researchers believe that they can and cannot do. I don't say that you need to agree with them but I think reading them should give you a good primer on why some researchers are not as excited as HN users are.

[1] https://dl.acm.org/doi/10.1145/3442188.3445922

[2] https://aclanthology.org/2020.acl-main.463/

[autophagian]

Thank you!

[Version467]

The recent post from Stephen Wolfram[1] is pretty good as an introduction, but I haven't seen any super comprehensive material that tries do dissect all the interesting behaviour we see in the really big llms. For that just reading the relevant papers themselves has been pretty fruitful for me. Some of them are actually very well written, even if you aren't used to reading scientific papers. I can recommend the Sparks of AGI paper[2] and the toolformer paper[3].

Obviously there's much more out there, those three things are a pretty good read.

[1]: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

[2]: https://arxiv.org/abs/2303.12712

[3]: https://arxiv.org/abs/2302.04761

[pcthrowaway]

At this point, anyone using the term "stochastic parrot" is just giving credence to my personal belief that humans are also stochastic parrots.

[runald]

A more apt term would be delusional parrot, applies both to the LLM and to everyone else who thinks GPT is the second coming of Jesus.

[autophagian]

I mean, its probably a component (or an approximation of a component) of what we do, at some level. Christ knows I've felt like a stochastic parrot when I'm zoning out 3 hours into a meeting and someone asks me a question out of the blue. I probably have a smaller context window at those points than GPT3 does...