[weitendorf]

My company tried to build something like this pre-TUI as a tool-AI-IO dag dispatcher. The biggest mistake I made was thinking that people would have no problem figuring out how they could translate their work or define multi-step automations, and focusing on the orchestration and sandboxing thinking that was the core, when it was really figuring out how to get the onboarding UX/complexity to not feel daunting or more trouble than it was worth.

Eventually for my own work, I discovered that the context management and runtime was more like a stream or active service mesh than a dispatching / one-off processing problem, most others' were too. Then all my prompts would degrade across model versions or providers, and I realized that actually setting the context for the tasks and keeping track of it all was a ton of work and something I had to do everytime as an actual user, but never when I was testing or demoing it on existing data.

Curious how you're testing your work and if you've managed to avoid the problems I ran into. I need to permute across the same set of workloads/configs you mention (and maybe more) for my next set of work so I'd be very interested in sharing or collaborating on the test infrastructure! At Google I did a lot of permutation testing using https://github.com/cloudprober/cloudprober and was going to start using it sometime in the next couple weeks. It exists basically one layer above the workload content/targets so it's probably compatible with everything except the test client/driver you're using.

[aleqs]

I'm my case a workflow is basically an active/living graph of nodes/sub-tasks. One node can process a task (with all relevant context) and create multiple fan-out tasks, or it can add additional context/requirements and pass it along to another node. The message/task passing is all implemented as queue - nodes subscribe to messages/tasks addressed to them and execute them, producing more tasks (or zero new tasks). For each task there is a context and a parent task/context, as well as a key/value store of all tasks and their context. Each agent/node gets instructions injected into their prompts that tell them how to look up parents tasks/context as well as how to output new tasks.

There is also a feedback loop - a node can fail to process a task, and pass the reasoning/context for that back to the parent or another node - this might result in a new adjusted task replacing the failed task, or it might require human intervention.

[weitendorf]

How do you test it across different workloads and are you running it in a datacenter or cloud provider?

I forgot to mention it but the other major problem I underestimated was giving the permission to potentially spend lots of money to AI calling each other in ways I didn't have a good way to monitor, and didn't want to actively watch. So I wanted to set budgets and have them get passed to children, and realized that meant I had to build a pretty complicated billing/scheduling system with a way to keep the part of it with all the permissions and money safe from the AI doing AI stuff on its own, and set up NAT and firewalls and all this other stuff.

If every child can loop back up to its parent, and everything can run stuff from the Internet, and make expensive resource decisions, and get restarted if it fails, then it might not ever converge on being done, or get infected or just mess up and spend a lot of money. I ask about the testing matrix/driver you're using because that's where I realized there was a lot of work and cost involved in getting that part working well enough to run real workloads.

[aleqs]

I have a 'node/container' abstraction at the infra/engine layer which is essentially either a cloud VM or a local podman container. The engine/infra layer can spin up more of these as needed. I have a relatively beefy dedicated machine for working with AI, which is where I do most of the testing.

I aggressively try to keep costs down so the workflow DSL I have supports configurable limits which can be set at the $, token, or time dimension , at task, workflow and agent/node levels, with some same defaults. I have a pipeline which keeps LLM API pricing data up-to-date, and I use AI to estimate total costs before runs and manually approve those.