[swagonomixxx]

I think this is one point that I don't see being made often enough. ChatGPT is exceptional at retrieval of information, but if it starts passing tests like the MCAT and LSAT and so on, I feel this is more that our testing is just crap, rather than ChatGPT being "smarter" than humans.

[jfengel]

Being a doctor does require a lot of just plain information retrieval. Human bodies have a kajillion parts interacting in a kajillion-squared ways. And we usually don't get to look directly at most of the parts; we have to infer what's going on from very indirect evidence.

This isn't one of those "memorize nothing you can look up" situations. You just have to have it at your disposal before you can do actual medicine.

There may be a day when a computer can do most of the memorization scutwork and you can get back to the real work, but we're not there yet. It's not the test's fault that it's testing medicine as it's practiced today.