[galaxyLogic]

LLM can write programs in any programming language it knows about. So how about askinng it to write a shell-program that does the tool-calls on the client?

You might want to run in some kind of sandbox to prevent the LLM from taking over the world, security is an isssue. But apart from that why not make the LLM write shell-programs instead of relying on JSON etc. ? Shell-scripting is the language for controlling the OS.

[dragonwriter]

Not always shell—Python, often a subset, is common—but a single sandboxed coding-running tool as the way to run all of the other things that would otherwise be their own top-level tools themselves seems an increasingly common approach.

[galaxyLogic]

I think Agents need user-accounts on your machine, and for us an easy way to configure the permissions each agent has, and then we dont' have to worry about sandboxing so much. This approach was pioneered by smart-phone OSes like Android.

Once permissions are safe and secure and easy to set up and understand then using shell-programming as a general interface between agents and your PC/OS might be a good option.

Come to think of it I recently read that Microsoft is planning to produce an AI-oriented OS. Maybe agents with user-accounts is something they are aiming for, to solve the problem discussed in the article.