[xmaayy]

I think it's more likely that 5 came out because if it ever saw the answer, 105, before, it was split into the tokens [10][5] of which it only 'remembered' one. Or the numbers were masked when training (something that was done with BERT-like models) so it just knew enough to put a random one in

[Nihilartikel]

That seems likely and fair.

What moved me to post is that that kind of silly answer is the exact sort of shenanigans that I would pull if I were cast as the control group in a Turing test.

I already do such things winkingly when talking with my preschooler to send him epistemic tracer rounds and see if he's listening critically

[doctor_eval]

> epistemic tracer rounds

that's the best phrase I've heard all year.

I do this all the time with my kids too, but I think of it more as fault injection.

[ramraj07]

Some of the funniest jokes I tell are retellings of jokes from obscure comedians of the past. How is this AI any different from me?

[mellosouls]

You enjoy the joke.

[visarga]

AI enjoys "low error rate". They have to, otherwise they would not exist.

AIs that don't lower the error rate are abandoned, AIs that score well are replicated and improved. It's evolution at work, but they have to enjoy (optimise for) lower error rates in order to even exist.