[fnordpiglet]

Enlighten us then how a generative AI model behaves when confronted with data outside its training space? Where in the model does it allow for the vector space to extend dynamically based on some other process to adapt to new regimes never seen before? Or does it necessarily construct its response by sampling the vector space, and in the case of transformers, apply attention / self attention to boost / dampen dimensions based on the semantic context? Extrapolation means being able to extend your decision space into new areas through synthesis and creativity, interpolation means walking within the trained vector space of the model. Clearly generative AI models as implemented today can’t extrapolate and always interpolate.

I think confusion comes from the idea that you can take a regression or expectation and extend it into the future and is that extrapolation. It isn’t - it’s interpolation still. You’re interpolating between a and a’ using the same function. Extrapolation takes the new regime and data and your existing training and adapts a new behavior. We don’t really understand how humans do this, and we don’t have any machine learning models that can.

To be clear, again, I’m not poopooing ML or generative AI. I think it’s the most powerful thing we’ve created with computers so far. But it’s far from general intelligence, even if it’s a necessary part.

[famouswaffles]

>Enlighten us then how a generative AI model behaves when confronted with data outside its training space?

It behaves just fine.

>I think confusion comes from the idea that you can take a regression or expectation and extend it into the future and is that extrapolation.

Congratulations, you've just defined extrapolation. Someone is definitely confused here but it isn't me.

Of course you can make any claim about what something can or can't do when you make up your definitions.

There are many many clear examples of a language model extrapolating. Rather than accept this, you've opted to conjuring up vague and meaningless definitions and distinctions on the fly.

This is so simple to see. Untestable Definitions are meaningless. Please give us a test of "extrapolation" that all humans can perform and let's see how the Language Model does. You won't be able to but by all means, give it a go.

>Extrapolation takes the new regime and data and your existing training and adapts a new behavior.

Great and Language Models do this.

[dragonwriter]

> I think confusion comes from the idea that you can take a regression or expectation and extend it into the future and is that extrapolation.

That's the literal definition of extrapolation, so I think the confusion is coming from your side.