[derbOac]

This was interesting to me but mostly because of a question I thought it was going to focus on, which is how should we interpret these tests when a human takes it?

I wasn't sure that the phenomena they discussed was as relevant to the question of whether AI is overhyped as they made it out to be, but I did think a lot of questions about the meaning of the performances were important.

What's interesting to me is you could flip this all on its head and, instead of asking "what can we infer about the machine processes these test scores are measuring?", we could ask "what does this imply about the human processes these test scores are measuring?"

A lot of these test are well-validated but overinterpreted I think, and leaned on too heavily to make inferences about people. If a machine can pass a test, for instance, what does it say about the test as used in people? Should we be putting as much weight on them as we do?

I'm not arguing these tests are useless or something, just that maybe we read into them too much to begin with.