[mdp2021]

Let us be more clear: there is no structural jump, no architectural overcoming of the original fault.

(Edit: and on a similar point, structural properties such as having static ntetworks, as opposed to continuously learning and improving architectures (such as us), will reveal that there is still road ahead.)

[Barbing]

Maybe more fair then would be: “I've worked with these systems for four years now and while they _have_ meaningfully improved in that time frame, they’re not perfect and remain fundamentally flawed in various ways.”

You prompt less. You need not inject search results into the context window yourself, a window much larger than years ago. You get code that’s already been run successfully once instead of finding an obvious show stopping bug yourself.

The technology is not a brand new one that fixed everything wrong with the old one, no, but not sure I would’ve noticed your comment if it had been such a bland observation. I genuinely assume good faith here… will say am tempted to assume the standards of someone posting such a thing might be impossibly high. Glad to be having a fun conversation instead of getting your grades on my work product or something :)

[orwin]

If you go to a LLM without harness, GP original point in completely right.

LLMS by themselves are still shit at math, they still confuse weird correlation to causation every time (and sometimes in ways even a 9 year old would say "no, that's dumb"), and confuse original parameters very often.

I disagree with " "thinking" token generation being on the correct track just to 'no, wait' on an already correct conclusion", because i think that is an effect of the harness, not the LLMs.

80% off all the improvements since ChatGPT4 are in the harnesses, and the LLMs by themselves, while they improved in areas they already were good at (translation especially) did not fix any of they original issues (object permanence, calculusm correlation).

Just run old models in the playground and get them to play chess (maybe make a small custom harness if you feel like it), then replace it with a frontier model (i don't know if you still have API access without harness on US models, but if you don't try K3), you will see LLMs weaknesses were not at all fixed, even marginally. They're way better and not inducing bugs in the code, so that make them usable since Opus4.5 (anyone using them prior to that either had a greenfield project or like spending hours debugging).

[rf15]

I agree, harnesses is where everyone improved the most. Our internal experimental tool can now semi-reliably formulate small programs to assert the correctness of their theories, for example. Context length is still a weird factor that we haven't sensibly solved - if anything, the lesson learned was to keep the context as small as possible and do most of the true "thinking" in the harness and temporary generated code.

[user43928]

> 80% off all the improvements since ChatGPT4 are in the harnesses

That seems easily falsifiable by putting an old model into the current harness and comparing it to 5.6 Sol or Fable.

[Barbing]

Sounds very fair, thanks :)