[oersted]

Frankly, just Python. LLM generation is just a function call, fetching from a vector db is just a function call.

LLMs are hard to tame at scale, the focus is on tightly controlling the LLM inputs, making sure it has the information it needs to be accurate, and having detailed observability over outputs and costs. For that last part this new wave of AI observability tools can help (Helicone, Langsmith, W&B Weave...).

Frameworks like LangChain obscure the exact inputs and outputs and when the LLM is called. Fancy agentic patterns and one-size-fits-all RAG are expensive and their effectiveness in general is dubious. It's important to tightly engineer the prompt for every individual use-case and to think of it as a low-level input-output call, just like coding a good function, rather than a magical abstract intelligent being. In practice, I prefer to keep the control and simplicity of vanilla Python so I can focus on the actually difficult part of prompting the LLM well.

[spacemanspiff01]

What are your thoughts on dspy?

[oersted]

It has caught my attention, I keep hearing about it as a new industry standard, I keep meaning to try it.

The reason why I keep procrastinating it is that, again, experience has shown me that LLMs are not really at a point where you can afford to abstract away the prompting. At least in the work I have been doing (large-scale unstructured data extraction and analysis), direct control over the actual input string is quite critical to getting good results. I also need fine-grained control over costs.

The DSPy pitch of automagically optimizing a pipeline of prompts sounds costly and hard to troubleshoot and iteratively improve by hand when it inevitably doesn't work as well as you need it to out-of-the-box, which is a constant with AI.

But don't get me wrong, I know I sound quite skeptical, but I intend to keep giving all these advancements a serious try, I'm sure one will actually be a big upgrade eventually.

[NeutralCrane]

In my opinion, utterly useless and I put it in the same bucket as Langchain. Lots of grandiose claims but doesn’t actually solve any problems people have.

I think we are at a stage where people are so eager to build something around LLMs to become the next shovel-maker, that a lot of what is being built doesn’t actually serve anyone’s needs.