[coolKid721]

Proper usage of LLMs so you don't just flood them with useless context will just be custom tailored prompts that only include the pertinent context, with prompts saying how it's related to what you're looking for. I don't think there's a cheap way around it maybe on the plus side you can tune them using ai code. I think tools are really over used and over-rated and have had horrible experience with them, nothing beats just custom tailoring stuff and setting up a system around it.

What I do is use elixir pheonix, have a genserver keep track of the user state and I just include the related state in the request and just helper functions to generate the related prompts per type of state/context and append them wherever makes the most sense.

I think LLMs make most sense to be viewed of as singular atomic interactions where you have the whole input (prompt/context/data) and get a concrete output. Everything else just seems like people being lazy trying to avoid thinking about the best way of structuring it. Where you put the context/data and how you include it will vary per prompt or the specific atomic interaction, there is no standard rule each interaction is unique. You have to experiment and see what provides the best output for each kind of request. I'd read Anthropics prompting docs if you haven't it's very good. https://docs.anthropic.com/en/docs/build-with-claude/prompt-...

My way of thinking is just viewing every isolated LLM request as a unique function that is the prompt + llm = a unique function Context is just what data you pass into the function (prompt+llm+settings(temp, etc))(data) to get whatever specific output you want. The prompt includes prewriting user/system messages, system prompt, structured output stuff or whatever. Any single request might lead to 1 or 30 of these that feed back into each other. But yeah based on that it depends on just custom tailoring them for anything, it's pretty conceptual and intellectual but I find it fun but I don't think there's any easy way around it. Having the ability to have all your requests be stateful and modify what goes into the prompt based on the current user state (like genservers/elixir makes very easy) is a nice technical thing that helps though.