[dazzaji]

I’m still stuck on the first sentence "An LLM should never output anything but tool calls and their arguments” because it just doesn’t make sense to me.

Tool calling is great, but LLMs are - and should be used as - more than just tool callers. I mean, some tools will have to be other LLMs doing what they’re good at, like writing a novel, summarizing, brainstorming ideas, or explaining complex topics. Tools are useful, but the stuff LLMs actually do is also useful. The basic premise that LLMs should never output anything beyond tools and arguments is leaving most of the value of LLMs on the table.

[bsenftner]

I think the blog simply does not explain well. Consider the example of a text editor, the "tool calls" are text fragments generated by the LLM then embedded into text editor tool calls that place the generated text fragment into the text editor, performing cuts, pastes, and so on.

FWIW, I've done this and it works incredibly well. It's essentially integrating the LLM into the text editor, and requests of the LLM are more like requests of the text editor directly. The mental model I use is the editor has become an AI Agent itself. I've also done with with spreadsheets, web page editors, various tools in project management software. It's an incredible perspective that works.

[dazzaji]

Got it, thanks for clarifying! So if I’m understanding you right, you’re saying that all the generative stuff the LLM does—like creating text—basically becomes part of the ‘arguments’ the original post talks about, and then that gets paired with a tool call (like inserting into a text editor, doing edits, etc.). I was focused on the tool call not the argument content aspect of the post.

And it sounds like you’ve had a lot of success with this approach in an impressive variety of application types. May I ask what tooling you usually use for this (eg custom python for each hack? MCP? some agent framework like LangGraph/ADK/etc, other?)

[bsenftner]

I noticed fairly early that the foundation LLMs have the source code to most FOSS, as well as the developer conversations, the user discussions trying to understand how to use that software, and the documentation too. The foundational models have a good amount of training data of each popular FOSS app, and by examining the code and the developer comments, and then adopting their language style, the LLM practically takes on the persona of the developer. So I spent some time understanding the internal communications of each app, and my 'tool calls' are structured JSON of the internal structures these applications use, and my own code receives these structured outputs and I just replace in the application's running memory. Not quite so blind as I describe, some of the insertion of these data structures is complicated.

In the end, each app is both what it was before, as well as can be driven by prompts. I've also specialized each to have 4 agents that are as I describe, but they each have a different representation of the app's internal data; for example, a word processor has the "content, the document" in HTML/CSS as well as raw text. When one wants to manipulate the text, requests use the HTML/CSS representation, and selections go through a slightly separate logic than a request to be applied to the entire document. When one wants to critically analyze the text, it is ASCII text, no need for the HTML/CSS at all. When one wants to use the document as a knowledge base, outside the editor, that's yet another variant that uses the editor to output a RAG ready representation.

[dazzaji]

That system would make a tidy startup, especially if tightly integrated with an open source office suite behind the scenes (LibreOffice, OpenOffice, etc) and a generative AI native UX.

[dazzaji]

* I'd call it "VibeOffice".