[jesse_cureton]

It's similar - basically here a vector/tensor is an array of magnitudes across N dimensions. Whereas in (undergrad-level) physics you might have a 3- or 4-D vector for spatial dimensions and time, here the LLMs are embedding sequences of tokens into N-dimensional space where N is much much larger.

There's a pattern called "embedding search" - you precalculate a set of embeddings for a corpus of text. Then to do a search, you calculate embeddings for your search string. Then you can find the closest vector in that N-dimensional space, which finds you the semantically closest neighbor from the original corpus.

For an embedding search - the OpenAI Embeddings API gives you a ~1500 dimension output vector. When a LLM is working with input text as a vector, I am not sure what the tokenizer is actually feeding into the model. Hopefully someone else can chime in!

[iamerroragent]

This is really elucidating.

Thank you and your time for writing that out.