[jimmaswell]

> For things you're already highly skilled at LLMs can be handy but the overall gains, after all is accounted for, are not so clear.

Preposterous. I can tell 5.6 Sol to do something like "optimize this entire subsystem to have no ongoing memory allocations, fix these 5 bugs, implement these 2 features" and go work on other code or do something else entirely, and it does it all flawlessly. All things I could have done myself but nowhere near as fast and right the first time with such thorough test cases. We've reached the point where the AI is usually faster and always less effort overall compared to the old days, even if you have to send a few followup prompts.

[layer8]

> it does it all flawlessly

Serious question: How do you assess this? It seems to me that it would take very significant time to establish that conclusion.

[Lord-Jobo]

Validation is really hard and requires a great understanding of the entire relevant code. So to answer your question, they don’t know or asked the AI to do that too which has very obvious problems.

I find it hard to believe they would spend the effort and money making an AI do tasks like that, but are ALSO willing to spend the effort and time to properly validate the changes. I’m sure many do… but probably not a large percentage.

[jimmaswell]

In my case it's usually a video game in question, not a life support system, so I'm not poring over every line of code.

I mainly look at:

- The overall architecture to make sure it's in line with what I want

- The points where it interacts with certain other systems I'm concerned about

And I test the feature myself in addition to the test cases. Sol has proven to be outstanding at good test cases and having a coherent view on overall architecture integration. I basically never think to myself "this entire implementation is slop, I need to rip it out and refactor it" anymore.

I suspect the model's capacity to translate requirements into contextually appropriate test cases, and insistence on thoroughly covering functionality in tests, is a large part of its strength.

[bbg2401]

This is going to sound accusatory but I promise it isn’t.

Did you feel the same way about 5.5? The same general sentiment keeps being expressed with every model release. The prior generation is immediately cast aside with a vague “well yes, actually we didn’t mean it last time” attitude.

I think I even heard Theo the T3 guy suggesting that his work comes to halt if he can’t use the latest generation model, despite heaping significant praise on each generation previous and suggesting it does all of his programming tasks.

The boundless energy with which each model generation is said to be revolutionary and leaving everyone behind is tiresome even when incremental progress is real and measurable.

[jimmaswell]

Prior to 5.5 I thought the models were very useful but not trustworthy for certain tiers of tasks and needed more in depth verification. Frequent hand edits and times where it felt more productive to just finish something myself than keep rolling the dice on the model not subtly messing it up. But still great for targeted things.

5.5 on Extra High I found to be a huge leap in what it could tackle and how effectively but only up to a point. Still had common failure modes here and there but a big improvement

When I tried 5.6 I wasn't expecting more than an incremental upgrade, but it blew me away with how "smart" it seemed, how it completely sidestepped failure modes I'd been used to dealing with in 5.5, its success rate on the first try for wide-sweeping architectural tasks, and how I've basically never seen a bad hallucination make it through to the final output. It's the first time I've felt like if I left it with a task on the highest setting and it had any way to verify its output, and told it to keep going, it could solve almost anything. This could all be incremental upgrades in many different areas of the model+harness, but the cumulative effect feels like crossing that "escape velocity" threshold. The point where there are no more rough edges and it "just works".

I can't speak to others, but my feelings on all of these have been consistent since I tried them in the first place. I've also heard model providers draw people in at each new release with the model at full performance and then degrade it over time hoping users won't notice, which has seemed plausible anecdotally but I have no hard proof.

[orwin]

That's highly unlikely that this is done flawlessly. This morning I found a bug in our error handling, I asked Sol to fix it. He rewrote the way the specific component that failed called the exception. I asked it to retry and go deeper, inside the library. It solution was to ignore the part of the message that caused the issue. Which worked very well, but wasn't the root cause (it was a serialization issue. It's always a serialization issue). It couldn't go deeper, always stuck trying to catch the error rather than fixing it for real. We found the root cause, fixed it, and it was done, one line change and 15 lines of test, in one hours. Instead of four hours of nothing except spending tokens.

But truly I'm mad at the vscode harness. I used to be able to follow the 'thought' of the AI model and steer them when I saw a mistake, it's now way harder to do as multiple research tasks are done in parallel.

[jimmaswell]

Somehow I've never had an experience like that on Sol. I find it to never really "cheat" like that. I've had all my best results from Codex CLI though, maybe that's somehow different. I also don't go below xhigh for anything serious and if it's having an issue I escalate the thinking mode.