[Jensson]

Yes, loss minimization quickly gets to the correct implementation of arithmetics since the primitives of neural networks are just math operations, so training it to add or multiply two inputs into an output is very easy. This is so easy and obvious that you run it to test that your neural network implementation works, if it can't figure out arithmetics then you have done something wrong.

LLMs fails to figure out that this is what it has to do, instead it looks like it has a ton of specialized rules to handle arithmetics that results in a lot of errors in the output and are extremely expensive to run.

[wruza]

So the networks you mentioned aren’t LLMs? Why is that a correct comparison then. Like blaming a human that they can’t jump like a cat or multiply like an arbitrary-precision library.

[Jensson]

> So the networks you mentioned aren’t LLMs? Why is that a correct comparison then

Because an LLM is a neural network and neural networks contains neural networks. There is nothing stopping it from having an embedded neural network that learned how to do computations well, except an inability to identify such structures and patterns well enough to train for it.

[jncfhnb]

Tokenizing ‘1735’ as a value of 1735 because you’ve seen a lot of math is probably the most difficult part.