[Isinlor]

If you look at the 3 sequences I gave you, can you guess following elements of the sequence?

We can create sequences, but guessing underlying patterns is a lot more difficult.

Humans will have very hard time if you go beyond around 10 operators in a pattern used to generate a sequence.

My guess is that their model will be better at it than me or you.

[YeGoblynQueenne]

Ah, I think I see what you mean: you are saying that because it's better than humans at predicting the next element in a sequence it's good at generalising. Is that correct, or am I misrepresenting your point?

[Isinlor]

Yes.

Basically there are two approaches to sequence prediction.

The traditional style, linear regression, ARIMA, RNNs etc. where you directly predict the next element in a sequence. The output is on the same level of abstraction as the internal values used in the model.

There is also the new-ish style where you predict symbols instead of predicting the values directly. You can predict symbols representing numbers or you can also predict a symbolic formula that can be used to extrapolate the values perfectly. This is the way humans do it.

And my point is that when you look at the symbols embedding they do have interpretable structure that model can use to generalize. And experiments seems to suggest that DNNs models are indeed generalizing.

[YeGoblynQueenne]

OK, thanks for the explanation. I think I understand what you mean. But this kind of generalisation takes very careful analysis to discern and I'm not convinced, yet. I'll be more easily convinced when I see something blatant, and n ≤ 10 is so far not there for me, even given the shift in what is predicted.

[Isinlor]

I still don't know what you mean by n ≤ 10.

Have you played around with the demo?

http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com...

You can input there a sequence and it will predict the expression and next elements.

For example, for the sequence: [3, 1, 8, 9, 18, 19, 37, 74, 92, 166, 332, 406, 738, 1476]

It predicts: u_n = u_(n−1) + (u_(n−3) (n mod 3)) + u_(n−4)

It's a really decent guess, and if I give it one more element (from 14 to 15 elements) it gets it correctly.

[YeGoblynQueenne]

>> I still don't know what you mean by n ≤ 10.

I mean the values of n_pred and n_op in tables 7 and 8. They go up to 10.

I haven't tried the demo. Maybe I'll give it a go if you say it's so good.