Hacker News new | ask | show | jobs
by famouswaffles 1095 days ago
>Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain.

alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?
2 comments
mjburgess 1095 days ago
see, just one reference in the paper: https://arxiv.org/pdf/2302.08399.pdf
link
famouswaffles 1095 days ago
they can answer those

https://medium.com/@nathanbos/prompting-better-theory-of-min...

link
mjburgess 1095 days ago
Yes, by changing the words

The whole point is that irrelevant word permutation should not "turn on" or "turn off" this capacity.

That you can "prompt engineer" your way to the answer shows that the prompt engineer knows the answer and can "use the right search terms" to find it.

link
famouswaffles 1095 days ago
"But the bag is transparent" is not "irrelevant word permutation" and neither is the additive question that spurs the correct resolution. And it certainly isn't random.

a human that isn't paying attention could fail the question too which is kind of the point i'm making.

There's no way a model that can't model protein structures does this - https://www.researchgate.net/publication/367453911_Large_lan...

link
YeGoblynQueenne 1095 days ago
>> alright let me humor you for a bit.

That's real class, right there.

link