[JumpCrisscross]

“AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years.”

Yup. Short-term metrics juice. Actual comprehension and cognition falls. This seems to be the case across the board, including with adults.

I’m genuinely optimistic that there is a way to make AI helpful in education. I just don’t think we’ve found it yet. (We certainly haven’t demonstrated it.)

[sisve]

> reduces completion time by 30%

This is probably the big problem, or at least one of them.. If you use less time on learning, it will probably be harder to remember what you learned also. We need to spend some time to make it stick

[solid_fuel]

The behavioral issue I see is that LLM users tend to immediately reach for an LLM and do their thinking in concert with it.

This tempts users to approach problems by first feeding them into the LLM and then simply following the route the LLM lays out, which does improve task completion time for tasks that the LLM can simply regurgitate, but it stops the user from developing the actual critical thinking skills that school is supposed to teach.

[cheesecakegood]

It’s not just critical thinking skills, it’s also that there’s a big difference between recognition/following instructions, and recall/generating your own memories of an approach. But most students don’t recognize the difference. In other words, “following the route” is a big part of the problem - it doesn’t engage the brain the same and isn’t representative of real world use, and having something explained well doesn’t mean you can in turn explain it well yourself (the more revealing test of internalized true understanding)

[sisve]

Can agree on that.

The description of the paper also said:

AI users who maintain similar homework completion time as non-AI users experience small learning losses.

This was a surprise too me. I would have thought otherwise.

Would love to see some evidence about if more or less people fall behind and have worse results. In my head the AI should be able to get the weakest students a bit highere.

[JumpCrisscross]

> In my head the AI should be able to get the weakest students a bit higher

I think the evidence so far is all students lose learning and cognition, but the brighter students lose less.

[ApolloFortyNine]

I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before. I still recall some parts of Physics taking a while to click, and often having to reread different sections of a textbook to try and understand the what and why behind something.

The biggest issue is a child has to want to do that, since they also could just ask the AI for the answer and then go back to playing video games. End of the day past age 13 or so I just don't see legislation making any difference, they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting.

[JumpCrisscross]

> I think AI could (and by some students probably already is) be used to help a student better understand the material, and faster than you could before

I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults.

> biggest issue is

We genuinely don’t know what the biggest issue is. We just know it doesn’t work. There is zero quality evidence for AI helping with learning or cognition in kids or adults. (Happy to be proven wrong. This is a fast-moving and big field.)

> they'll find a way past any law blocking them from using AI. Like a lot of education it'll probably come down to parenting

And community. Rich towns restrict devices in school, monitor use at home and thus will have less of an issue with AI exposure.

[ApolloFortyNine]

>I think so too. But we haven’t demonstrated we’ve found how, in kids or in adults.

Ask chatgpt or claude, on their highest model (probably unnecessary but I'm sensing a vibe) to explain a simple linear algebra problem, and if you don't understand it, ask about what part you don't understand.

And if you truly believe it made something up, prove it.

This is seriously the easiest thing to prove out there, you can see for yourself in the next 5 minutes.

[solid_fuel]

> And if you truly believe it made something up, prove it.

You seem to be assuming that the issue is around factual correctness, and that may be the case but the evidence we have so far doesn't support jumping to such a narrow cause.

Is the poor performance because the LLMs are frequently wrong? Unknown.

Is it because the LLMs are sycophantic? Unknown.

Is it because the chat interface is a poor one for learning? Unknown.

What we do know is that students who rely on LLMs learn less and perform worse in the long term. And that alone is enough evidence to support a ban. If better tools come along in the future and are shown to aid learning, then the ban can be re-evaluated.

[JumpCrisscross]

Sure. I already know linear algebra. If it’s a new branch of mathematics, this is a terrible way to learn.

Again, the research points almost exclusively in one direction when it comes to learning and cognition around AI. You’ll solve more problems more quickly but wind up learning and thinking less.

[ApolloFortyNine]

Honestly, what do you think a teacher does that an llm can't? If you want to learn you absolutely can ask an llm how to _solve_ x and explain the steps and why.

My leaving out the word solve seems to have led some of you astray, I apologize.

Again the problem is you have the option to solve your problem and move on without understanding it. That does not mean you can not use the tool to understand the problem and how to _solve_ it.

I live in fear that instead of learning how to use the tool, some might just vote to ban the tool.

[JumpCrisscross]

> what do you think a teacher does that an llm can't?

I don’t know! It’s an interesting question. All we know is it does.

> That does not mean you can not use the tool to understand the problem and how to _solve_ it

It doesn’t. But we have no evidence it can.

We have lots of evidence of people thinking they’ve learned something, taking a benchmarking test, and being found wholly deficient compared to folks who worked through a textbook, went to a class or even solved problems off YouTube videos or instructional websites.

[wieiw]

lol people like this guy prove AI psychosis is legit.

If you can’t figure out what the value add of a human teacher is then.. fkin lmao. It’s well beyond simply transmission of information.

The best teachers have passion - that passion is infectious. I was lucky enough to experience that and it grew my curiosity.

LLM’s provide no such equivalent.

[skydhash]

> to explain a simple linear algebra problem, and if you don't understand it, ask about what part you don't understand.

The goal is not to understand a linear algebra problem. The goal is to learn how to solve it using lessons and techniques taught beforehand. Aka not to get a fish, but learning how to fish.

[ApolloFortyNine]

I'm sorry the wording of my post didn't match what you wanted.

Type in "Explain how to solve a simple linear algebra problem" into the AI of your choice instead.

[JumpCrisscross]

> Type in "Explain how to solve a simple linear algebra problem" into the AI of your choice instead

I’m more interested in seeing how someone who teaches themselves with this approach scores on a standardized exam of linear-algebra competence.

[skydhash]

> Type in "Explain how to solve a simple linear algebra problem" into the AI of your choice instead.

I’ve seen this particular philosophy in college where the student focus exclusively on passing exams. They would memorize notes and past exercises. The focus is on solving a particular set of exercises instead of understanding the concepts. Change things slightly and they’re lost.

That may not matter in college where you can focus on a few disciplines and half-ass the rest. But everything in lower stages is truly foundational.

[cyberax]

A crucial part of learning is struggling with understanding and overcoming problems by yourself. AI removes that part.