[brotchie]

I'm still on the fence about agent frameworks, they have their place, and it depends on the nature of the agent: e.g. "Low latency, return a good enough response in 3 seconds, vs. working for 3 hours on a problem."

BUT, if you boil it down, an agent really is context building, making an LLM call, executing requested tool calls, parsing the final model output, returning it to some frontend. There's extensions like memory, async tool calls, etc, but not THAT complicated from a traditional software engineering perspective.

Everyone seems to want to build their agent framework. But if you're tasked with building an agent, I've found it much easier and more maintainable to just build 1:1 code for THAT agent: most of the abstractions you get from an agent framework purely get in the way and obfuscate core agent logic.

You end up being forced to use the abstractions chosen by the agent framework, which sometimes are a mismatch for what you're actually trying to do.

[peterbell_nyc]

For me the heart of an agentic system is NOT using agents (except when you really have to). Components of a working system include: - Pipelines/recipes to describe multi-step flows (deterministic, agentic and HiTL steps), loops, conditionals, exit-on's for max loop iteractions, etc - The logistics to actually run the model and HiTL steps reliably across multiple agent worker pools - Management and delivery (and security/governance and permissioning) of thick skills with code to do as much as possible - Context management so the right agents have the right context for the right sessions at the right time - Project management - ability to store and access tickets, dependencies, track progress, restart stuck ticket claims, etc - Transcript saving, memory features and dreaming/compounding capabilities so the agents continue to learn from each session - o11y for understanding whats happening, tracking costs and usages, etc - Evals and auto-tuning of prompts so you can go cross model provider and also lock to a model version so you can do an ROI on each model version upgrade - Sandboxes for running the actual model sessions

Don't need to get it all from one vendor, but that feels to me like the toolkit and for most use cases I'd argue: - Don't limit yourself to a single model provider (anthropic, openai, etc) - Own your context - Own your compounding

[sdesol]

Can you comment more on

> Context management so the right agents have the right context for the right sessions at the right time

I'm going to do a show HN tomorrow that explains how you can give your agents years of experience. The basic idea is, you would commit in your repo or download manifests (JSON files) that can be converted to "Brains" (SQLite databases). Each brain can have its own properties.

For example, I provide a "code intent" analyzer (instructions for AI) that says when analyzing a file, extract this metadata. For the code intent analyzer, I have the AI extract a single sentence purpose for the file. So if you execute:

gsc rg cache --db code-intent --fields purpose

you get all matches for 'cache' plus the matching file's purpose like "Modify file to update caching strategy". This is how the agent can tell if the file is talking about cache vs. whether this file is what you should change if you want to update the caching strategy.

So for what you described, you can have a brain for different stages of a task. It can be as simple as, in the planning stage, make sure you do this if you need to touch this file.

I am working on a rust-blast-radius brain that uses `syn` + AI generated metadata to help you understand "what if I changed this file, what would be affected". With the rust-blast-radius brain, the AI can summarize the types of files that will be affected without having to open the file based on what has been changed or discussed.

So you can have a rule like, if I make changes to a Rust file, make sure to do a blast radius analysis so we don't forget to consider something.

Does this align with what you are looking for?

[andai]

(I'm not the guy but) That's funny, I had the same idea the other day. Keeping summaries of files. Haven't tested that yet.

Another thing I've been thinking is how, most parts of a file are not relevant to the whole system.

Like there are parts where they intersect, and those seem to be the most important ones for capturing the big picture. You wanna be able to see the entire "skeleton".

So I thought the summary maybe shouldn't be English but it should be a subset of the code — the subset that's relevant to the rest of the program.

`grep import` gets you 90% of the way there.

[sdesol]

If you include the following:

https://github.com/gitsense/chat/blob/main/base-state/analyz...

In your chat with AI, include the above file and let it know what your requirements are and I can create the analyzer and include it.

You can also think of my tool as data prepping tool. So if you have a clear prompt the AI can review the file during analysis and remove all unnecessary code so the extracted metadata will the stripped text which you can use search against.

[andai]

> If a developer wanted to change X, would these keywords help them find this file?

I think the best way to generate these is with a sub-agent. Tell it to try and solve a problem that involves editing this file, and see what it starts grepping for.

This ties in with this idea that the tools and designs should be what comes naturally to the LLM, i.e. what it's already been trained on. And the most straightforward way to do that is to let it reach for it.

Like when you reach in the darkness for an object. Where your hand lands is exactly where it should be.

[sdesol]

My solution has a natural self improvement loop. Once you have finished a task, you just ask the agent "If you had more information, how would you have finished the task sooner and/or better?" This was how I came about the rust blast radius brain.

I need to modify OpenAI's Codex agent to support slash commands that can help humans better guide agents, and I needed a solution with the least impact. They don't accept contributions so I need to plan for syncing with the upstream.

[internet101010]

I do file summaries as well. Basically a knowledge base commit hook + agent that creates/updates a file containing a summary and list of its dependencies, followed by updating an index to include it. Super useful for creating system migration scopes and just managing context in general.

[kristjansson]

Obscuring core logic is the most egregious part of most agent frameworks. One needs a clear view of what, exactly, is being sent to the underlying language model, and what's coming back. Everything in an 'agentic' application is realized as a sequence of tokens or a call to a provider eventually. It should be clear and obvious from ~all layers of the app what that's going to look like.

[cpard]

Most framework vendors don’t have an incentive to make things less obscure. The agent framework is free/open source and they make money primarily from selling observability products for agents. Even if they don’t intentionally obscure things, they just don’t have the motivation to optimize that part.

[pjmlp]

Unfortunately agent orchestration frameworks feels like the second coming of BEPL, and the incentives are all wrong.

[throw1234567891]

Have a look at pi.

[tomrod]

Whst about pi do you like?

[peterbell_nyc]

Pi is a nice multi-agent wrapper. I use it to wrap my OpenAI max plan calls and my API calls. It takes care of some of the agent plumbing - still need sandbox, orchestrator, compounding, context, evals, etc but it's a nice component.

[chuckadams]

Any particular plugins you'd recommend? For orchestration I gave pi-subagents a try, and didn't care for it, ended up with hung agents sticking around forever and I wasn't even doing anything terribly fancy. Claude's subagent control is annoying and clumsy, but it works.

[krawczstef]

FYI - Burr is designed with recursion in mind, i.e. the ability to kick off Burr within Burr. So you could have an action that is managing several Burr subgraphs... Or you write your own management layer here.

[lukebuehler]

Arguably, if your agent needs a lot of custom logic to drive the agent loop it isn't an "agent" at all. At best it is an agentic workflow, or rather a workflow with some LLMs calls in between.

I think that's why agent SDKs feel like the wrong abstraction. If you are writing a workflow, use a workflow engine (Airflow, Temporal, etc), and call some LLMs with a small LLM library. If you need a "real" agent, use a full-featured agent harness, like Codex or CC or Pi or whatever, then load it up with all the tools, skills, mcps that it needs, and let it rip.

Incidentally I've been building a full featured agent harness that runs inside durable workflow engines [0], but it is designed _not_ as an SDK but rather as a standalone, full-featured harness with an API.

[0] https://github.com/smartcomputer-ai/forge

[dd8601fn]

I’ve said something similar about dozens of frontend frameworks. It’s massive abstraction and convolution for some future payoff that’s obviously never going to happen.

But sometimes people just need something to do, or something fun to play with, and “the next guy” rarely matters that much… so who cares that you’ve saddled them with the result of your paid playtime?

[andai]

So before AI I had the experience, more often than not, that it would take me longer to figure out how to use someone else's thing (or get it to do some particular thing, which often turned out to be impossible), than to just make my own.

And that was before I could just ask the computer to make it for me!

But most people seem to be the other way around. They'd rather deal with abstractions and boilerplate instead of writing the actual code.

[mbreese]

I'm somewhat in agreement. I like building 1:1 code for that specific agent.

Where I'm starting to question this is maintainability. When I come up with a new technique or way of doing something in my new agent, how can I update an older agent. Do I want to update the older agent?

But, I get what you're talking about w.r.t. building for the exact problem at hand. For example, I'm guessing that Apache Burr has support for a plugin-able vector RAG system (or at least it will if it doesn't now). That's great, but I want my RAG system to add documents to the context and keep them as part of an updated system prompt with some very specific tweaks that happen as part of that process. This is a bespoke way of working with an existing concept (RAG) that doesn't lend itself to using any specific framework.

In my use-case, bespoke is the way to go. But then I'm still stuck with having to make engineering choices for updating older agents. So, I see your point.

[cyanydeez]

Agents are a way to de-bloat the context. The way LLMs function, you absolutely need to find the sweet spot for a given task, and if the primary LLM has to go through a bunch of failures to find a working function, those failures are better contained in an agent and disposed of.

Obviously, you could have a different LLM like a "angel" that prunes a primary agent of the context it doesn't need, but I think the realistic KV cache problem is will determine the optimal structure: you want the work do be done in the most efficience KV cache (context-reuse) as much as possible.

There's definitely more to it than just spawning agents.

[freakynit]

Yep.. same. I build my own agents... all use-case specific. Keeps the code super minimal, and avoid unnecessary complexity. I have tried a few of these, but nop.. no help.. only more work (and issues).

[hilariously]

Couldn't agree more - tried to convince a business that doubling down on OpenClaw wasn't going to solve problems except for some 0-1 stuff, and that almost immediately they'd run into roadblocks because most of the product wouldn't serve their use case.

4 months of mostly spinning their wheels later they launched a really lackluster OC product that's effectively DOA.

[tcdent]

OpenClaw is an application, not a harness. Yes, it contains a harness, but it is a complete product.

When building an agentic workflow there are enough primitives that rewriting them from scratch every time makes zero sense.

What is a tool? How does the LLM understand the tool? Formatting a native function into a serializable input/output pattern makes sense to generalize and that does not need to exist repeated in everyones application code.

We use libraries to interact with the APIs themselves; nobody would say writing a spec-compliant API client was poor practice. Agentic harnesses are just one layer above: I need to call the API and I need to do it with certain expected conventions.

[hilariously]

Sure, but that seems liked two different points.

One, obviously yes OC contains a lot more than a harness, but my point was that it was too much for their use case and constrained their choices, not enabled them, and that choosing the right layer of abstraction is important.

There's good indirection/abstraction and there's ones that do not serve your use case, eg what was obviously day one regarding Langchain.

[pianopatrick]

I like to think of it as "AI prompting algorithms". Like instead of just this prompt gets this result it's A prompt then B prompt the C prompt gets a result.

And just like when people were trying to figure out which sorting algorithm made the most sense, we are all just trying to figure out which prompt algorithms with which models lead to good results.

[chuckadams]

The most interesting evolution of my agent workflow over the past year is that I've dropped all the language gimmicks: I used to use particular emoji to mark parts of code as hints to the AI, I structured planning docs in very rigid language for different types of instructions, and generally optimizing my language for machine consumption. That's all gone now: all my comments, directives, plans and so on are just plain and clear English, nothing more. They're direct and unceremonious, but no longer bullet points that were nearly caveman-speak. It just works better that way.

[vanuatu]

my job rn is just building agents

the hard part about building agents isnt the framework it's discovery, context, traditional engineering, handling the last mile

there are some invariants like the loop, tools, observability, guardrails, monitors etc...

[brotchie]

100% agreed, the "this is what an agent looks like to write" is the wrong pitch for a new agent framework.

The better pitch would be, "this is how easy observability, guardrails, monitoring, deployment, evals, versioning, A/B testing are with our framework." What the agent code looks like is somewhat incidental.

[peterbell_nyc]

This this this!

Anyone have something they genuinely like for all of this? For now I'm rolling my own, but I can't believe I won't find a better OSS alternative soon...

[agentdev001]

Nvidia Openshell solves most of the hard problems I've run into while building stuff in this space.

Observability is, for my purposes, solved by a given framework supporting OpenTelemetry.

Guardrails is where I've gotten the most value of openshell being a neat package. Agent workload scope is written as policy in openshell, and capability is backed by openshell handling all execution.

Monitoring/deployment/versioning is helped as well, depending on how agents/runners are slotted into the system. Deployment namely is quite well supported- openshell has kube/helm bits that are experimental atm, but seem like a logical approach imho.

Evals and a/b testing isnt something ive explored in depth, considering that agents with composable tool sets + frontier models are beyond my expectations already.

[brotchie]

We need the equivalent of the MEAN / LAMP stackronym for agents.

[elijahbenizzy]

Right I think this is why we made it unopinioated to a fault. Burr doesn't really do these things rather it just provides an orchestration framework. So it's pure BYO functions, classes, components, etc...

[fxwin]

The advantage of frameworks isn't that they make it easier to write the actual agent, it's tooling + observability + ... Even Langchain, for all the (deserved) criticism it gets made this very clear very early: It might be easy/easier to write your own chatbot from the ground up, but what happens if you have to add observability/tracing? Being able to just add one environment variable and instantly have a UI where i can nicely go through all of my traces with basically 0 additional effort is something a hand rolled solution just can't really compete with

[freakynit]

This only becomes relevant if your execution graph is complex/big enough. Otherwise, all it takes is less than 30 minutes to add telemetry to all needed points. Doing manually also gives you better control on what you really want to track (to save costs).

[trollbridge]

It’s painfully obvious that you can just open your coding harness and… tell it you’d like to make an agent. They’re simple to write.

[peterbell_nyc]

The agent isn't the hard part - it's the orchestration, skills, research systems, adversarial reviews, dreaming/compounding, context management and all the rest. Plus all the annoying hygiene tools to "poke the agent that got a clear prompt and decided to just sit there and wait for no good reason" and "delete the remote branches that the prompt told all the agents to delete but some of them forgot to":)

[elijahbenizzy]

This is where it's nice to have some guardrails -- coding agents work etremely well with limitations.

[toddmorey]

Anthropic seems to agree with you as more recent Claude updates have it just building task specific harnesses as needed.

[bko]

Take a simple workflow. You have a query it goes to a classifier. The classifier determines what workflow it should route the request to.

Then you have a general workflow that has a set of skills (prompts) and tools. And that could be recursive.

So if you do something like "rename this file" you have to build up a workflow like:

[classifier]

what's the workflow -> rename

[rename workflow]

list files (tool call)

figure out relevant predicate (LLM)

convert predicate into a filter query give the context of the files (LLM)

figure out what you want the new name to be (LLM)

create the request body and hit the tool

approval workflow

formatting

It's a lot to manage and orchestrate and that's just one simple example. You'd like want to use the same building blocks to delete a file or move it. Even to know the right concepts is difficult as we're a bit deluded on whats going on in the background of these modern AI apps like Claude and GPT that do a lot of this stuff for you

[krawczstef]

yep - here's something cool an end user wrote with Burr - https://github.com/msradam/phoebe

Burr just helps you, the engineer, to really control the primitives. Then adds some cool features you don't have to think about -- like observability :)

[agentifysh]

100% this

you dont need a framework