[NitpickLawyer]

The paper is here - https://arxiv.org/pdf/2603.19461

This, IMO is the biggest insight into where we're at and where we're going:

> Because both evaluation and self-modification are coding tasks, gains in coding ability can translate into gains in self-improvement ability.

There's a thing that I've noticed early into LLMs: once they unlock one capability, you can use that capability to compose stuff and improve on other, related or not, capabilities. For example "reflexion" goes into coding - hey, this didn't work, let me try ... Then "tools". Then "reflxion" + "tools". And so on.

You can get workflows that have individual parts that aren't so precise become better by composing them, and letting one component influence the other. Like e2e coding gets better by checking with "gof" tools (linters, compilers, etc). Then it gets even better by adding a coding review stage. Then it gets even better by adding a static analysis phase.

Now we're seeing this all converge on "self improving" by combining "improving" components. And so on. This is really cool.

[binarymax]

I disagree that evaluation is always a coding task. Evaluation is scrutiny for the person who wants the thing. It’s subjective. So, unless you’re evaluating something purely objective, such as an algorithm, I don’t see how a self contained, self “improving “ agent accomplishes the subjectivity constraint - as by design you are leaving out the subject.

[NitpickLawyer]

Sure. There will always be subjective tasks where the person who asks for something needs to give feedback. But even there we could come up with ways to make it easier / faster / better ux. (one example I saw my frontend colleagues do is use a fast model to create 9 versions of a component, in a grid. And they "at a glance" decide which one is "better", and use that going forwards).

OTOH, there's loads you can do for evaluation before a human even sees the artifact. Things like does the site load, does it behave the same, did anything major change on the happy path, etc etc. There's a recent-ish paper where instead of classic "LLM as a judge" they used LLMs to come up with rubrics, and other instances check original prompt + rubrics on a binary scale. Saw improvements in a lot of evaluations.

Then there's "evaluate by having an agent do it" for any documentation tracking. Say you have a project, you implement a feature, and document the changes. Then you can have an agent take that documentation and "try it out". Should give you much faster feedback loops.

[alsetmusic]

> Things like does the site load, does it behave the same, did anything major change on the happy path, etc etc.

I asked Claude to build a web app to run locally polling data from the LAN. It fought me for four rounds of me telling it that the data from the api wasn’t rendered on the page. It created tests with mock data, it validated the api, it tested that the page loaded. It was gaslighting telling me that everything worked every time I told it that it didn’t work. I had to tell it to inspect the dom and take screenshots with Playwright to make it stop effing around. I don’t think it ever would have found the right response on its own.

Even after deliberate intervention, it regressed a few rounds later and stopped caring that tests failed. Whatever, I don’t treat it as anything more than a sometimes-correct random output machine.

[latentsea]

The thing you're missing is harness engineering.

[ranyume]

In science there are ways to surface subjectivity (cannot be counted) into observable quantized phenomena. Take opinion polls for instance: "approval" of a political figure can mean many things and is subjective, but experts in the field make "approval" into a number through scientific methods. These methods are just an approximation and have many IFs, they're not perfect (and for presidential campaign analysis in particular they've been failing for reasons I won't clarify here), but they're useful nonetheless.

Another thing that get quantized is video preferences to maximize engagement.

[Atomic_Torrfisk]

Im sorry, this just sounds like hypespeak. CAn you provide samples?

> once they unlock one capability,

What does it mean to unlock? Its an llm nothing is locked. The output is a as good as the context, model and environment. Nothing is hidden or locked.

[IncreasePosts]

Maybe unlock means "recognize and solve a problem with an order of magnitude fewer tokens than the first time you did it". The same way humans might spend a lot of time thinking about a certain problem and various ways to solve it, but once they go through that process, and then recognize it again, they don't need to go to the same process and jump right to the solution.

[seertaak]

I'll have a stab at this. I'll start with an attempt at justifying the remark that an agent which is a good coder will be good at other tasks.

1. Coding is, as a technical endeavour, relatively difficult (similarly for mathematics). So a model which performs well on this task can be expected to easily handle also-technical-but-slightly-easier tasks, like understanding (musical) harmony theory or counterpoint -- for much the same reason that human programmers/mathematicians/scientist don't struggle to understand those "easier" theories.

2. Reinforcement learning augments a base models ability to excel in something else that's "difficult", namely to "look ahead" and plan multiple steps in advance. That's literally how the training algorithm works, generating multiple paths at once, and rewarding intermediate steps in those paths which succeed in attaining the goal. And that skill, too, is extremely useful in other domains. An AI agent which learns that to break a problem into sub-problems, and then tackle each in turn methodically -- it stands to reason that it can apply that to, say, a business plan.

Note: 1 & 2 are not independent, nor are frontier models' excellence in these domains magical: it ultimately boils down to the availability of massive datasets (in particular for coding) and totally objective metrics (in the case of mathematics: solved math problems). That's the key ingrediant for reinforcement learning to be so effective.

So: the skills are transferrable because they're difficult, and require lots of planning. That models are so good at them is a fluke, and in a parallel world where humans created git repo after git repo of business plans, it might be that which we lean on to teach a reinforcement learning algorithm how to "reason" and "plan".

Now let's turn our attention to the "synergies" aspect, which I agree with. Let's say your agentic model, which is already excellent at reasoning and planning, acquires a new or improved capability which allows it to search the domain space, calculate, etc. much better than before -- this capability can now bear upon the plan, or be factored into the plan. For example, the model might be able to say "I don't need to worry about this particular subproblem for now; I can rely on my "mathematica" capability to deal with it when I absolutely need."

Or to put it differently: monkeys, like humans, are able to use (rudimentary) tools. They'll take a rock, and use it to crack open a coconut (or whatever). But a human being, with far superior reasoning and planning abilities, takes that tool, and uses it to make an even better tool -- and the result after many iterations of this process is civilization as we know it, while monkeys are still stuck trying to crack open nuts with rocks.

[lukebuehler]

Agree. It's code all the way down. The key is to give agents a substrate where they can code up new capabilities and then compose them meaningfully and safely.

Larger composition, though, starts to run into typical software design problems, like dependency graphs, shared state, how to upgrade, etc.

I've been working on this front for over two years now too: https://github.com/smartcomputer-ai/agent-os/

[iamandoni]

> Larger composition, though, starts to run into typical software design problems

I've been seeing the same thing. Where agents are great solving the immediate task, but as changes compound they run into software & architectural design problems. I created https://github.com/andonimichael/arxitect to help at least have coding agents self reflect on their software design. But I really like your approach to self-modification and improving the agent itself instead of just teaching it another skill in it's context.

[dataviz1000]

I’ve been working with Claude Code to create copies of itself using git worktrees, run an iteration and then update its instructions. It can reverse engineer every website I tested it on. I kept updating the instructions then started asking Claude to update itself. Then asked if could figure out who to iterate unsupervised. https://github.com/adam-s/intercept?tab=readme-ov-file#the-s...

[whattheheckheck]

So what are software packages now a days other than precomputed subsets of capabilities. Like a mesh that data gets pushed through to produce what? What are the optimal subset of prebuilt programs to accomplish any task?

[lukebuehler]

It's a tradeoff. Technically, you need very few programs, you can let an agent do everything and coordinate everything. But that is also inefficient, it's slow and uses a lot of tokens. So you allow the agent to build tools and coordinate those tools, just like we humans do. However, with agents, the threshold of pain is much higher, we can let agents do thing's "manually" where humans would build automations much sooner.

[ghywertelling]

Exactly. Tools like grep, ls are also in the same category. Even in algorithm, we have techniques like memoization and Dynamic Programming which allow us to speed up things. Why should LLM fill up its context by "manually" doing what wc or ls does for you deterministically ?

[whattheheckheck]

Oh wow, what do you think of karpathys autoresearch? Feels like this is just that? Gotta openclawify it?

[sbinnee]

I guess this paper is part of ICML coming soon this June. I hope to see a lot of cool papers.

[FrustratedMonky]

IF they are self modifying. Is there also a big risk, that they cause a bug, to dumb themselves down, break themselves. How do they get back? Are they able to restore a backup of themselves, if a self modification is bad.

Or, are there two. One is modifying the other, observing results, before self applying.

[derek1800]

Yes this is how hyperagents work. They keep track of their different attempts to see which results in success. It does require tasks that can be measured.

[naasking]

Agents need the ability to code but also to objectively and accurately evaluate whether changes resulted in real improvements. This requires skills with metrics and statistics. If they can make those reliable then self-improvement is basically assured, on a long enough timeline.

[derek1800]

This is how hyperagents work. They Have the ability to measure improvement in both the meta agent and task agents. There approach requires task agents to tackle tasks that can be empirically evaluated.

[alexpotato]

>You can get workflows that have individual parts that aren't so precise become better by composing them, and letting one component influence the other. Like e2e coding gets better by checking with "gof" tools (linters, compilers, etc). Then it gets even better by adding a coding review stage. Then it gets even better by adding a static analysis phase.

This is the exact point I make whenever people say LLMs aren't deterministic and therefore not useful.

Yes, they are "stochastic". But you can use them to write deterministic tools that create machine readable output that the LLM can use. As you mention, you keep building more of these tools and tying them together and then you have a deterministic "network" of "lego blocks" that you can run repeatably.

[ACCount37]

Yep. Behavior composition. If you train an LLM to do A and to do B, separately, chances are, it'll be decent at A+B despite not being trained for the combination.

[alansaber]

The whole theme of llm dev to date has been "theres more common than not" in llm applications

[testaccount28]

because submarine piloting is a going-under-water activity, improvements in holding one's breath can lead to faster submersibles.