[fishtoaster]

There's some good ideas and points in here, but this bit threw me:

> # We tried this > In July 2025 we went full lights-off

Isn't it pretty well-accepted at this point that the models underwent a step-change in usefulness around fall 2025 / spring 2026? I know that I was able to start handing agents whole features after that, but not before.

I feel like any perspective/experience on "what agents can/can't do" from before that period is... maybe less than relevant to the modern era. TFA calls it out a few sections later with "But surely the models have gotten better since then", but then just writes off any improvement. That does not match my experience.

[2001zhaozhao]

I had a bit of this impression when reading the post as well as the authors' product website. A lot of it does seem to be stuck in 2025.

For instance I think their post "long-context isn't the answer" on their website post straight-up isn't accurate, and gives me the impression they are just extrapolating previous performance to new models. In my experience, Opus 4.6 and newer have worked very reliably for long context to me (i don't perceive any intelligence drop at 700-900K tokens). Yeah it's extremely cost-inefficient, but it works.

[chmod775]

> i don't perceive any intelligence drop at 700-900K tokens

I have countless examples of this for Fable 5 and Opus 4.8 at just half that many tokens. Sometimes they can't even remember the correct workflow even if they're merely an orchestrator and have been going through the same 5 step loop multiple times for pretty much their entire context window.

Wall of shame with an Opus and a Fable oopsie from just yesterday: https://i.imgur.com/znSJ3OE.png and https://i.imgur.com/DLOHQ1B.png

One of them is particularly egregious because it uses a CLI tool to step through the stages that literally tells it what to do next. How do you fuck this up? Needless to say that tool was revised to force the agents to acknowledge everything happened, but I'm sure it will find a way to eventually mess up anyhow.

And orchestration should be the easy part. It's pretty much just writing and adjusting workflow files. Don't get me started on code quality - but I realize that's subjective and my bar is unusually high. There's a reason I have them run adversarial verifiers along multiple axis before their output is even worth my attention.

[hansvm]

FWIW, I notice the intelligence drop drastically with long contexts with Opus 4.6. It's barely usable for anything intricate and long. That long window is good for _something_, but it's not as good as a short window.

[Terretta]

for synthesis. read the entire whatever in one gulp and boil it down or plan what to do about it.

not for steps.

[hansvm]

Oh yeah, that's a use case I love long contexts for. "Here's a day of prod logs, what's broken?" It's an incredible utility, but the qualities much less than with shorter contexts, and you see it much more apparently when you do similar coding tasks with progressively larger windows.

[eru]

> Yeah it's extremely cost-inefficient, but it works.

And competition is fixing that as we speak.

[tcoff91]

Opus 4.5 was a massive step change in capability for sure. The opinions of anyone who hasn't bothered experimenting with these things post-Opus-4.5 are not worth a damn.

They still are far from perfect but they're massively helpful.

[json_hero]

read. https://github.com/humanlayer/advanced-context-engineering-f...

[superfrank]

I just listened to a podcast last week with the author (Dex Horthy) on as the guest. Humanlayer (his company) is constantly trying to push the limits with agents and he's very into using them.

He's well aware of exactly how good the models are today and I'm sure that if he thought they were good enough now to make the dark factory idea work, they'd be trying again.

[RamblingCTO]

The obvious statement would then be: 2025 we went all in, 2026 still nothing improved substantially. I also agree with that statement, none of the frontier models handle drift any better. Shotgun surgery and all that. You're waving him off by waving him off, but provide no arguments whatsoever. Is your experience substantially different?

[dhorthy]

my perhaps controversial take is that opus 4.1 was smarter than 4.5 for complex engineering work, but 4.5 was faster and "squishier" - it responded better to simpler prompts, it read between the lines of user input better, and that this was really important to converting new users quickly

[json_hero]

Did you even read further on? The section right after that addresses the step changes in current frontier models https://github.com/humanlayer/advanced-context-engineering-f...

[dhorthy]

it is absolutely wild to me that this keeps floating to top comment thread when the guy very clearly did not read even the first half

[fishtoaster]

As I mentioned explicitly in the comment you're replying to: yes, I did read that section. Did you not bother to read my whole comment before telling me to read more?

[paulhebert]

> I feel like any perspective/experience on "what agents can/can't do" from before that period is... maybe less than relevant to the modern era

This part of your comment is confusing.

The article talks about agents past this period. Yet it seems like you’re saying this article isn’t relevant because it doesn’t?

[svara]

OK, but having read that section you'll see how

> just writes off any improvement

Is not what that section contains?

There's a more nuanced point about capabilities improving in some ways but not others (ability to one-shot apps vs. maintenance of existing code over time).