Interpolation vs extrapolation? It's very simple distinction and fundamental. Look at ARC-AGI-3 for empirical evidence, if that's important to you. My point however is purely an a priori one.
> so the proof to the unit-distance problem was on the manifold, given it was outputed by a LLM?
By design this must be the case, even when you account for stochastic sampling i.e. 'temperature'. All it's outputs are a highly-dimensional combinatorial interpolation (I'm talking about GPTs here)
That's probably why Claude is very good at producing plausible nonsense rather than the often correct response of 'I don't know'.
The argument stands.