[williamtrask]

Poor performance is more likely due to how transformer neural networks view numbers. It memorises them like words instead of modeling their numerical structure. Thus even if it’s seen the number 3456 and 3458, it knows nothing of 3457. Totally different embedding.

It’s like a kid memorising a multiplication table instead of learning the more general principle of multiplication (related: this illusion is why big models are so popular. Memorise more stuff.)

Paper (NeurIPS/DeepMind): https://arxiv.org/abs/1808.00508

[Isinlor]

Take a look at this paper:

Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600

If you look at embedding visualization it is very clear that the model learns order of numbers.

(Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... )

There is also:

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177

Again, looking at visualizations the model very clearly grasps the structure of the function it models.

[pfortuny]

Modulo 97 (the arxiv paper). That is what they do.

It is quite easy to grok operations modulo 97.

[YeGoblynQueenne]

The "Deep Symbolic Regression" paper reports very poor generalisation results that break off after a small n (where n is the number of tokens in the predicted sequence). It works some of the time for n = 1 (predicts the next token) but accuracy drops off for n = 10. No results are reported for N > 10 as far as I can tell in the "Out of Domain Generalization" section (which is the meat and potatoes of the "generalization" claim).

tl;dr they can sometimes generalise to the next 1 to 10 tokens (digits or operators), but no more.

This kind of short-term "generalisation" on OOD data is standard in neural nets trying to approximate symbolic regressions or things like grammars etc as far as I know.

I do like they use 'Out of Domain" rather than "Out of Distribution" as a target though. That makes more sense.

[Isinlor]

I don't think you will find any human that will extrapolate sequence generated with more than 10 operators. And longer input sequences are actually easier to handle - fig 1. the right most graph.

If you think you can do better than their program then:

Seq1: [0, 1, 2, 3, 6, 7, 13, 26, 32, 58, 116, 142, 258, 516]

Seq2: [2, 2, 3, 5, 10, 12, 22, 44, 54, 98, 196, 240, 436, 872]

Seq3: [3, 1, 8, 9, 18, 19, 37, 74, 92, 166, 332, 406, 738, 1476]

Their program is able to guess correct continuation with one more sequence element.

SHA1 hash for verification: bef5e213340f91258b3b9a0042c9c083dd91cb80

[YeGoblynQueenne]

I don't think I understand what you mean. Aren't all the sequences on the Online Encyclopedia of Integer Sequences created by humans? We clearly have the tools to extrapolate sequences from examples, rather than just eyballing them and trying to guess them. For instance: we have maths. So I must have misunderstood your meaning?

[Isinlor]

If you look at the 3 sequences I gave you, can you guess following elements of the sequence?

We can create sequences, but guessing underlying patterns is a lot more difficult.

Humans will have very hard time if you go beyond around 10 operators in a pattern used to generate a sequence.

My guess is that their model will be better at it than me or you.

[YeGoblynQueenne]

Ah, I think I see what you mean: you are saying that because it's better than humans at predicting the next element in a sequence it's good at generalising. Is that correct, or am I misrepresenting your point?

[nicholast]

The cool thing about math applications is just how easy it would be to generate synthetic data. That these large language models haven't attempted to supplement their gigabytes+ scale data sets with such is an oversight.

[williamtrask]

Or you could just use a 50cent calculator.

Note, you’d need to train such a model on data teaching it about the relationship of every number to every other number when run through every function. Yes, infinite synthetic data, but you’re just memorising stiff you can already generate

[not2b]

Or build a model that has "peripherals". Oh, I'm being asked to do math. Let's put it in my calculator app. Everything doesn't have to be in one uniform network.

Evidently the brain works that way: the cortex is built on top of older components, so it doesn't have to figure out basic metabolism the same way it has to learn to identify people.

[plutonorm]

It's recently been shown that even though the numbers are represented with different tokens, the network learns to form an internal representation that understands the progression from one token to the next.

[nikolayasdf123]

The idea that each number has to be inside ones Brain or Neural Network or Token is plainly wrong.

Network has to grasp the "abstract" number, but it clearly did not grasp that concept.

[Isinlor]

How would you test if it grasped the concept?

[plutonorm]

https://arxiv.org/pdf/2201.02177.pdf

This paper shows fairly conclusively that the network 'groks' modular addition.

[pfortuny]

Modulo 97.

This is what it is. Not "general arithmetic".

[catach]

Being able to extrapolate to numbers that were not in the training set, perhaps? At least that'd be a basic part of the requirement.

[Isinlor]

Sure:

Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600

(Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... )

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177

Both of these models can generalize to numbers it have not seen.

[YeGoblynQueenne]

As far as I can tell from a quick heuristic perusal, the "Generalization Beyond Overfitting" paper reports "generalisation" _on the validation set_. That's not particularly impressive and it's not particularly "generalisation" either.

Actually, I really don't grokk this (if I may). I often see deep learning work reporting generalisation on the validation set. What's up with that? Why is generalisation on the validation set more interesting than on the test set, let alone OOD data?

[eutectic]

That depends on the tokenization scheme.

[moffkalast]

> this illusion is why big models are so popular. Memorise more stuff

It's all just a compressed lookup table that can handle in-betweens.