[bena]

> As for teaching to the test, what's the real downfall of doing that?

This comment is a perfect encapsulation of AI hype.

Recitation is not competence. It doesn't matter if Bobby can say "4" when I say "What is 2 + 2?". What matters is if Bobby can add.

[ryathal]

Test questions are written to test varying levels of competence from basic recall to true mastery. It's partially why so many people hate word problems, they don't actually understand the concepts. Many others just can't read.

[Hizonner]

Test questions can be written to assess true mastery, although it's not so easy as you're trying to make it sound... especially if you want the test to take a sane amount of time, be consistently gradable by different graders, and have any kind of reasonable signal to noise ratio. But sure, you can get there, or at least go a long way toward it.

If you try to actually create tests that way, you will discover that the institutions and political atmosphere around you will punish you harshly. It's "unfair" to give a test that you can't pass by memorizing things, you see. And the people who will ultimately make the decisions about what you're allowed to do with your test are rarely going to care much about actual mastery, and often wouldn't be able to recognize mastery if they did care.

In practice, standardized tests are always going to be easily gamed, so if you make people's and institutions' rewards dependent on them, you will end up diverting more time, energy, and other resources away from actually teaching mastery and into meaningless gaming.

[ryathal]

Standardized tests spend the money and time to get those quality questions that are consistently graded. It's not some random folks making minimum wage coming up with questions.