[majormajor]

Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs".

And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.

But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.

I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.

[geysersam]

They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...

I wonder if that's also how I'd behave if I had a reinforcement learning harness around me that dived hard on and punished me for every small mistake.

[denverllc]

That is not overengineering, it’s creating job security:

> Funny story no one will believe, but it’s true. A good friend of mine joined a startup as CTO 10 years ago, high growth phase, maybe 200 devs… In his first week he discovered the company had a microservice for generating new UUIDs. One endpoint with its own dedicated team of 3 engineers …including a database guy (the plot thickens). Other teams were instructed to call this service every time they needed a new ‘safe’ UUID. My pal asked wtf. It turned out this service had its own DB to store every previously issued UUID. Requests were handled as follows: it would generate a UUID, then ‘validate’ it by checking its own database to ensure the newly generated UUID didn’t match any previously generated UUIDs, then insert it, then return it to the client. Peace of mind I guess. The team had its own kanban board and sprints.

https://news.ycombinator.com/item?id=48061235

[layer8]

> They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...

That’s not overengineering, it’s plain nonsensical, because presumably it doesn’t compare it to all IDs generated in the past. Which, if you wanted to do that, you’d use a database with a uniqueness constraint, in case you don’t already have that anyway.

Fixing this lack of reliable common-sense awareness seems to remain elusive for LLMs.

[geysersam]

The ids were used in a limited context and it compared to all the other ids generated in the same context. Yes, absolutely nonsensical.

I find it interesting that presumably the LLM was trained on millions of legit examples where people generate uuids, but still it decided to do something different and unexpected, because it "reasoned" itself into thinking it was necessary.

[solarkraft]

> even the lower-end models can find the things they didn't think of first when inspecting the already-written output

I use Deepseek V4 Pro for my hobby project (an OpenCode client). The economics will obviously be different at work.

It’s worse than the GPTs, but indeed, if you focus it on reviewing its own code (essentially spending more reasoning and changing perspective), it’s also quite capable at improving its own approaches.

The GPTs are better at general architecting, but I think a lot of the performance gains also came from my more careful prompting (“okay, this is a hard problem, let’s think this through …”) to make using the more expensive model worth it. I use them and they are worth it on subsidized rates, but not at API prices. Since my Codex sub ran out I sometimes miss the models, but it really hasn’t devastated me.

[torginus]

On my own time I still like the 'chat' format of me writing a short prompt and having the model turn it into a few dozen lines of code, or make a simple refactor (like turn a field into a method parameter.

I wasn't impressed with Chinese models (I tried Deepseek V4 Pro/Flash, and GLM5.2) the amount of questionable code and mistakes and misunderstandings meant the money saved and faster speed of models didn't translate to faster progress for me.