[ClassAndBurn]

Any custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required.

Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.

[Ancalagon]

How is custom engineering the tooling from scratch for every task the best path forward? There will always be some engineered tools that are better than others even when re-made by the same models - not to mention the cost of starting from scratch every single time just seems wasteful with current token spend.

Does "don't re-invent the wheel" not apply to agents for some reason?

[Jensson]

> Does "don't re-invent the wheel" not apply to agents for some reason?

Correct, the purpose of agents is to become the ultimate wheel so we humans can stop needing to reinvent the wheel. For it to do that it needs to know how to reinvent wheels.

And, just to be clear, "don't re-invent the wheel" is just for small teams, as a whole humanity needs to re-invent the wheel all the time to adapt to new situations. For every wheel your team shouldn't reinvent some other teams main job is to reinvent that wheel.

[dpe82]

Humans (exhibiting "general intelligence") are tool builders; virtually all our capabilities stem from our ability to create and use tools - often extremely specialized to a task. Why would an artificial general intelligence be any different?

[Zababa]

"don't reinvent the wheel" isn't a law of physics and I think is mostly said by people that never designed anything with wheels.

[idiotsecant]

Bespoke tools can be simpler than general purpose ones

[notjackiechan]

I think there is a chasm to cross. the model's training to be aware of the harness it is in, at the same time building probes and observation tool can help it to cross that chasm.

Still seem too soon for a model to have that ability to build a harness on its own, and swap its session to another harness in the same environment. Like a snake shedding its skin, but in this case its harness.

[deepfriedbits]

Indeed. And you can make this case about any tooling at all that is model adjacent.

[jvanderbot]

Including by extension all programs, operating systems, or eventually hardware, I suppose.

[deepfriedbits]

Yep. That was the entire subtext from the drop in IBM's share price yesterday.

[re-thc]

> Sol Ultra style is the path forward

That’s called brute force so not really.

[cyanydeez]

I seriously doubt this, especially in a world in which there's not just one model.

This makes sense if the models some how become unified.

[pixl97]

I guess this is kind of an information theory thing. As a model converges AGI would a generalized harness ability appear?

[satvikpendem]

Codex and Claude Code already have the workflow feature which basically does this on demand, so it's getting there.

[vessenes]

Except in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.

[ClassAndBurn]

It's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to?

We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.

[vessenes]

Let's maybe say not experienced enough / insufficiently RL'ed then -- 5.6 Sol did not reach for a harness solution like this when it got only 13% or son AGI-3 recently. I agree it's interesting to find the point in the prompting when it could 'tip' and do this. I have no instinct for where that point is, except that it must be somewhere, because I bet the Schema Harness was not hardcoded.

[couscouspie]

Letting the provider decide for the harness is a terrible idea in my eyes. Outsourcing harnessing is giving up control over the AI and equivalent to abandoning your sovereignty. It is a regression to a pre-enlightenment era.