[kolinko]

This is a recurring sentiment but flawed, I think.

First of all, neural nets do nit return averages per se. They construct space between the points and extrapolate outside of the points. So even if a point was not in their training data, they will be ok, in many situations, to acknowledge it.

Or in other words - LLMs don’t average. They construct world models. A novel thing that fits their world model will be accepted no prob. A thing that doesn’t may still be accepted but with challenges.

The same is true though for humans, including scientists. There is a saying that science moves one grave at a time - because often prev gen of scientists needs to die off for a new idea to take root.

Or in yet other words - even if llms produced averages, an average of a discontinuous set can lie outside of that set. And the set of all human ideas is very much discontinuous.

[krona]

> They construct space between the points and extrapolate outside of the points.

They don't. They interpolate between the points on a manifold.

[kolinko]

Manifold vs space - yes, I meant manifold, english is not my first language and I missed this word.

The argument stands.

[krona]

Interpolation vs extrapolation? It's very simple distinction and fundamental. Look at ARC-AGI-3 for empirical evidence, if that's important to you. My point however is purely an a priori one.

[memoriyato3]

so the proof to the unit-distance problem was on the manifold, given it was outputed by a LLM?

is the proof to the Riehmann hypothesis also somewhere on the manifold and we just need to prod the LLM with the right prompt so it locates the point?

[krona]

> so the proof to the unit-distance problem was on the manifold, given it was outputed by a LLM?

By design this must be the case, even when you account for stochastic sampling i.e. 'temperature'. All it's outputs are a highly-dimensional combinatorial interpolation (I'm talking about GPTs here)

That's probably why Claude is very good at producing plausible nonsense rather than the often correct response of 'I don't know'.

[kolinko]

What you’re saying here makes no sense.