Imagine a calculator that computes definite integrals, but gives non-sensical results on non-smooth functions for whatever reason (i.e., not an error, but an incorrect but otherwise well-formed answer).
If there were a large number of people who didn't quite understand what it meant for a function to be continuous, let alone smooth, who were using such a calculator, I think you'd see similar issues to the ones that are identified with LLM usage: a large number of students wouldn't learn how to compute definite or indefinite integrals, and likely wouldn't have an intuitive understanding of smoothness or continuity either.
I think we don't see these problems with calculators because the "entry-level" ones don't have support for calculus-related functionality, and because people aren't taught how to arrange the problems that you need calculus to solve until after they've given some amount of calculus-related intuition. These conditions obviously aren't the case for LLMs.
The TI-83 Plus had an equation solver that didn't actually do any solving, it would test lots of inputs and converge towards the correct answer. If it was a rational number it worked, but it couldn't do fractions so anything else and you'd only get an approximation when it hit its limit.
What do you think is the big difference between these tools and *outsourcing*?
AI is far more comparable to delegating work to *people*.
Calculators and compilers are deterministic. Using them doesn't change the nature of your work.
AI, depending on how you use it, gives you a different role. So take that as a clue: if you are less interested in building things and more interested into getting results, maybe a product management role would be a better fit.
Fundamentally nothing, but everybody already knows that you shouldn't teach young kids to rely on calculators during the basic "four-function" stage of their mathematics education.
Calculators for the most part don't solve novel problems. They automate repetitive basic operations which are well-defined and have very few special cases. Your calculator isn't going to do your algebra for you, it's going to give you more time to focus on the algebraic principles instead of material you should have retained from elementary school. Algebra and calculus classes are primarily concerned with symbolic manipulation, once the problem is solved symbolically coming to a numerical answer is time-consuming and uninteresting.
Of course, if you have access to the calculator throughout elementary school then you're never going to learn the basics and that's why schoolchildren don't get to use calculators until the tail-end of middle school. At least that's how it worked in the early 2000s when i was a kid; from what i understand kids today get to use their phones and even laptops in class so maybe i'm wrong here.
Previously I stated that calculators are allowed in later stages of education because they only automate the more basic tasks; Matlab can arguably be considered a calculator which does automate complicated tasks and even when i was growing up the higher-end TI-89 series was available which actually could solve algebra and even simple forms of calculus problems symbolically; we weren't allowed access to these when i was in high school because we wouldn't learn the material if there was a computer to do it for us.
So anyways, my point (which is halfway an agreement with the OP and halfway an agreement with you) is that AI and calculators are fundamentally the same. It needs to be a tool to enhance productivity, not a crutch to compensate for your own inadequacies[1]. This is already well-understood in the case of calculators, and it needs to be well-understood in the case of AI.
[1] actually now that i think of it, there is an interesting possibility of AI being able to give mentally-impaired people an opportunity to do jobs they might never be capable of unassisted, but anybody who doesn't have a significant intellectual disability needs to be wary of over-dependence on machines.
There's a reason we don't let kids use calculators to learn their times tables. In order to be effective at more advanced mathematics, you need to develop a deep intuition for what 9 * 7 means, not just what buttons you need to push to get the calculator to spit out 63.
If there were a large number of people who didn't quite understand what it meant for a function to be continuous, let alone smooth, who were using such a calculator, I think you'd see similar issues to the ones that are identified with LLM usage: a large number of students wouldn't learn how to compute definite or indefinite integrals, and likely wouldn't have an intuitive understanding of smoothness or continuity either.
I think we don't see these problems with calculators because the "entry-level" ones don't have support for calculus-related functionality, and because people aren't taught how to arrange the problems that you need calculus to solve until after they've given some amount of calculus-related intuition. These conditions obviously aren't the case for LLMs.