[dathinab]

It guaranteed can't have all of your code in the LLM context, or even all of your file (in case of longer, through not even quite long files).

Through it could do stuff like go through your repository(ies) and generate embedding for sections of it and then have a vector database + retrieval argumented generation (RAG) system.

[threecheese]

A lead from one of the GH Copilot-adjacent companies was interviewed recently, and that’s precisely what they are doing. They generate embedding of “local” code based on AST (up the stack and sideways, if you know what I mean), and take into account runtime and library versions when doing inference. Sounded like a very interesting challenge.

[theshrike79]

Yea, that's the way they're going based on the low-memory models they're building.

Basically all the LLM needs to do is translate human writing to some format that a "normal" service can use. That can then leverage the existing spotlight system that's pretty decent at searching stuff on the phone anyway.

Then it'll report it back to the LLM which translates whatever format back to something humans can process.

Basically "less shit Siri"