[remexre]

Imagine a calculator that computes definite integrals, but gives non-sensical results on non-smooth functions for whatever reason (i.e., not an error, but an incorrect but otherwise well-formed answer).

If there were a large number of people who didn't quite understand what it meant for a function to be continuous, let alone smooth, who were using such a calculator, I think you'd see similar issues to the ones that are identified with LLM usage: a large number of students wouldn't learn how to compute definite or indefinite integrals, and likely wouldn't have an intuitive understanding of smoothness or continuity either.

I think we don't see these problems with calculators because the "entry-level" ones don't have support for calculus-related functionality, and because people aren't taught how to arrange the problems that you need calculus to solve until after they've given some amount of calculus-related intuition. These conditions obviously aren't the case for LLMs.

[Izkata]

The TI-83 Plus had an equation solver that didn't actually do any solving, it would test lots of inputs and converge towards the correct answer. If it was a rational number it worked, but it couldn't do fractions so anything else and you'd only get an approximation when it hit its limit.

[simonw]

I think we don't see these problems with calculators because we have figured out how to teach people how to use them.

We are still very early in the process of figuring out how to teach people to use LLMs.