[penguinten]

It uses the OpenAI embedding models to get around the context window problem. You can compress large amounts of text/code into just a few tokens.

[LunaticDragon]

Can you elaborate on this? What is the input to the GPT in that case? I was under the impression that GPT is given an array of tokens and it produces one token as its output.

[iamflimflam1]

Would love some more information on how to do this. Can you share details or point to a resource?

[cipherpro]

https://beta.openai.com/docs/guides/embeddings/use-cases

[joshka]

I'm curious if can share LoC -> token count metrics?