[zwaps]

Sol is a complete mess for me.

It only works on end to end tasks in fresh codebases.

Otherwise it cannot follow instructions, changes and deletes unrelated features or does sloppy work to mark a task completed while leaving a compromised codebase.

I could not get Sol to finish a feature in a complex code base without several loops of fixing and reverting

[Gareth321]

I agree. I tried hard to use Sol to its full potential but it makes sloppy mistakes. I suspect a part of this is the newly reduced context limit. Performance gets worse the more memory compactions have occurred. But I also subjectively notice a micro-focus temperament. Developing and deploying small parts of the project to a high degree while forgetting how that component fits into the larger architecture. Once it's complete it realises that what it built doesn't align well with everything else, then it rewrites the component (and other components). Rinse and repeat. I have to be much more prescriptive with Sol, which makes it much less useful for me. If I have to be prescriptive, I can use the Chinese models and achieve the same thing.

Opus provides genuine insights. Things I have not considered and have missed. More importantly, it allows me to skip most of the architecture decisions and requirements work. That's a genuine time saver for me.

[theplumber]

It happened to me as well but in a different direction: i.e adds non library code in a shared library. Another issue I with GPT is that it is chasing too much edge cases/security issues(I.e chasing ghosts).

However this makes it also a strong model because it fixes/solves problems that both Opus and Fable are incapable. In reviews it catches bugs that both Fable and Opus are missing to spot.

To me the “best of both worlds” is to research the problem with GPT sol, create a plan with Fable and dual review it with both Fable and GPT-SOL and implement it with GPT-SOL. You can see in the code reviews how many times both Fable and opus are sloppy and superficial while GPT-SOL just does its due diligence …I had several problems that Fable just gave up and it was GPT that helped it sort it out.

[zwaps]

But then Sol doesn’t really follow the plan or rather invents things it should add or cut on top of the plan. How does one deal with that?

[braebo]

I’ve spent over 100 hours trying to get GPT 5.6 Sol to resemble something useful as opposed to something actively detrimental. I concluded that it is not possible because it is fundamentally not intelligent the way Claude models are.

[xrisk]

Might be an indication that your task unit is too unstructured or your code base is a mess.

Is your actual code doing something complex or is this incidental complexity?

Relying on the model’s “intelligence” to patch over these issues hasn’t proven to be a reliable strategy for me. Of course, this might not apply to you, just my 2 paisa.

[Gareth321]

A well structured architecture, repo, and requirements document with concrete small deliverables can be competently delivered with cheap Chinese models. We rely on frontier models so that we don't have to spend several days/weeks on planning/architecture/documentation. That's their value proposition: superior intelligence. If they can't deliver that, they're useless at the current price.

[faebi]

I found out it really depends on the repo I'm working on, and it's not just about being a new/old repo, there's something else which I can't really grasp yet.

[derfurth]

Yes it’s very codebase dependent, and changes over model versions. In my case codex was so bad, until it suddenly became better than Opus at 5.5, which surprised me. I am sure it can be the inverse for some codebases.

The way I keep an eye on model performance is alternating between models for code review. It shows how much the model understands the codebase without disrupting my workflow.