[et1337]

Folks have been saying “things are different now, the agents are now compounding success instead of error” for at least a year now, but I just don’t see it. I was lucky enough to receive a weeklong $50k per head AI training from the people saying these things, and one of their few helpful concrete recommendations was to constantly clear context all the time, to avoid things going off the rails.

However, I think finding security vulnerabilities is one use case where it doesn’t matter. Tokenmaxxing is absolutely effective for that. We as an industry are in the middle of adopting very expensive, complex continuous fuzzers.

[jdiff]

Even modern frontier models benefit so hugely from careful context pruning, maintenance, and rewriting to erase mistakes that it's astonishing to me that there are no tools centered around it. The one tool that used to have such a feature, Zed and its retroactively-named Text Threads, has now stripped itself of it.

[RugnirViking]

this! the back-and-forth chat interface where you can edit only your own messages, and only then to get a new response, is a terrible one, but I think favored by vendors because it helps them fight in vain against prompt injection. Custom harnesses and stuff are nice but incredibly time consuming to set up when all I want to do is like, see what the agent is reading, and editing out some irelevant nonsense side quest it went on, or trim some massive log file it read which filled 90% of its context. Theres no inherent reason behind some caching gains that these things must be strictly chronological - a response it gave me previously does not have to be part of the context now

[hwillis]

"some caching gains" is a pretty huge understatement- snipping something out of the middle of the window requires rebuilding the entire context. Thats a shitload of tokens.

Afaik messing with the context also pretty reliably degrades performance still. The model responses reference things that no longer exist to it and it becomes more chaotic.

The real usefulness of parallel or sub-agents is not that they run at the same time, its that they isolate noisy or self-contained context away from the main window.

[MaxMatti]

I still feel that during agentic workload sometimes it would be nice to have the model identify it is veering off the main track, send out a "keep the cached states and tell me which they are" command to the inference server, do the side thing (such as handling an error that plopped up that has not that much to do with the main task) and return back to the cached state with just a comment tacked at the end to say "oh and btw I fixed DNS" instead of having the DNS debugging inside the context window now. Maybe other harnesses just steer the models more towards using subagents for such tasks and my pi is misconfigured. I can use the tree feature, but having insight into what's cached would be nice there.

[jdiff]

You can structure your context window to minimize the amount of editing you do further back. You usually only need to edit and correct the most recent response. It's little different from forking the conversation at an earlier point, and nobody warns about that being a sneaking footgun. There is still a prefix to cache.

[foota]

No reason other than you're giving user's a cost footgun? That's a pretty good reason.

[jdiff]

Local models benefit disproportionately from this kind of pruning and have no such footgun.

[Uzazo]

This is why I prefer Pi over all other agent harnesses. It has a tree view of each conversation and it's easy to move between branches.

[liquicity]

50k per head training and the largest takeaway was to clear context.. that is the "hello world" of using agents, insane.

[ButlerianJihad]

Have you tried turning it off, and then turning it back on again?

[olex]

Some companies only get to a "hello world" level with a new kind of tech via a 50k per head training. The organizations are setup in a way that people can't experiment or learn by themselves, it's really the only way.

[deadbabe]

$50k a head is cheap compared to the productivity gains, probably can push it to $75k

[tsunamifury]

Really. How?

Have you measured them beyond loc?

[kalcode]

It's true, but I see it happening. I’ve watched seniors with 30+ years of experience adopt them successfully without losing their classic rigor.

Personally, I get huge mileage out of LLMs, and yes, I care deeply about code quality, readability, and debuggability.

I've seen juniors absolutely rock with them.

And I've seen the exact opposite, where they just struggle to get good results.

In the end, I think the divide comes down to management experience. The people thriving are the ones who have led teams, especially teams of contractors, which is the best analogy for how you have to interact with an LLM.

Those folks know how to break down problems, provide the right context, and scope a task just enough to see the "contractor" succeed before letting them move forward.

On the other hand, individual contributors who are used to just grinding solo often struggle. They expect a one-shot miracle. They say, "Hey, my code is buggy, fix it." When the LLM inevitably hallucinates or steers them wrong, they give up. The results are completely different based on how you treat the tool.

They might just have a high quality of control and standards that it is hard to find that pattern with the LLMs.

I think fierce individual contributors are a lot more valuable in the era of llms as well. We as humans typically achieve better balance with new stuff when we allow backlash from new processes that start to trample on old ones without understanding AKA the Chester's fence.

Anyways, more of a ramble than my two cents.

[markatto]

I’ve been finding that very rigorous/technical people who are willing to adapt their workflows are very effective, whereas more “management” types are sometimes effectively pushing work off onto their code reviewers.

[atsaloli]

Who delivers a 50K per head training? Who pays for such a thing?

How many people were in the class?

I had a small training company, shuttered during COVID, and I used to charge 5K per day, for a group of up to 12 people. 5 days training = 25K. This is double, wow.

I would love to get back into training but getting enough volume to live and support a family on has been a challenge.

[seattle_spring]

> I was lucky enough to receive a weeklong $50k per head AI training

wow! That sounds like an unbelievable grift. Who were they such that anyone could possibly think that's a worthwhile investment?

[rezonant]

Isn't that like a day worth of tokens?

[codegladiator]

10 days

[xnx]

> weeklong $50k per head AI training

In Hawaii I assume?