[FeepingCreature]

I don't know what to tell you. They specifically showed PaLM novel jokes. You're effectively saying that the paper is either mistaken or fraudulent.

In my experience with language models, what they do cannot be reduced to madlibs. But that's obviously not an argument I can prove to you.

Can we agree that if the model can explain structurally novel jokes, then it must have some measure of true understanding?

[yldedly]

Understanding of what? What the joke is about? Then no, it has no idea what any of it means. The syntactic structure of jokes? Sure. Feed it 10 thousand jokes that are based on a word found in two otherwise disjoint clusters (pod of whales, pod of TPUs), with a subsequent explanation. It's fair to say it understands that joke format.

If you somehow manage to invent a kind of joke never before seen in the vast training corpus, that alone would be impressive. If PaLM can then explain that joke, I will change my mind about language models, and then probably join the "NNs are magic you guys" crowd, because it wouldn't make any sense.

[FeepingCreature]

Good point, coming up with a novel joke is no joke. There's a genuine problem where GPT is to a first approximation going to have seen everything we'll think of to test it, in some form or other.

Of course, if we can't come up with something sufficiently novel to challenge it with, that also says something about the expected difficulty of its deployment. :-P

I guess once we find a more sample-efficient way to train transformers, it'll become easier to create a dataset where some entire genre of joke will be excluded.