|
|
|
|
|
|
by notnmeyer
502 days ago
|
|
|
maybe i dont get it, but can we conclusively say that the gorilla wasn’t “seen” vs. deemed to be irrelevant to the questions being asked? “look at the scatter plot again” is anthropomorphizing the llm and expecting it to infer a fairly odd intent. would queries like, “does the scatter plot visualization look like any real world objects?” may have produced a result the author was fishing for. if it were the opposite situation and you were trying to answer “real” questions and the llm was suggesting, “the data is visualized looks like notorious big” we’d all be here laughing at a different post about the dumb llm. |
|
|
The gorilla is just an extreme example of that.
Albeit perhaps an unfair example when applied to AI.
In the original experiment with humans, the assumption seemed to be that the gorilla is fundamentally easy to see. Therefore if you look at the graph to try to find patterns in it, you ought to notice the gorilla. If you don’t notice it, you might also fail to notice other obvious patterns that would be more likely to occur in real data.
Even for humans, that assumption might be incorrect. To some extent, failing to notice the gorilla might just be demonstrating a quirk in our brains’ visual processing. If we expect data, we see data, no matter how obvious the gorilla might be. Failing to notice the gorilla doesn’t necessarily mean that we’d also fail to notice the sorts of patterns or flaws that appear in real data. But on the other hand, people do often fail to notice ‘obvious’ patterns in real data. To distinguish the two effects, you’d want a larger experiment with more types of ‘obvious’ flaws than just gorillas.
For AI, those concerns are the same but magnified. On one hand, vision models are so alien that it’s entirely plausible they can notice patterns reliably despite not seeing the gorilla. On the other hand, vision models are so unreliable that it’s also plausible they can’t notice patterns in graphs well at all.
In any case, for both humans and AI, it’s interesting what these examples reveal about their visual processing, which is in both cases something of a black box. That makes the gorilla experiment worth talking about regardless of what lessons it does or doesn’t hold for real data analysis.