[himata4113]

What I really started to notice is that SOTA models are really good at putting themselves out of the job.

We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges.

The more intelligent models get, the more people will offramp to cheaper solutions that get the job done. There's no real benefit to using a sota model when the accuracy is already 99% and I think that is the biggest danger to US labs.

[com2kid]

The upper end is all about coding. If I have terra on extra high write code, Sol will find a plethora of bugs and rip the code apart.

Anything else? Sure use a cheaper model.

[majormajor]

Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs".

And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.

But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.

I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.

[geysersam]

They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...

I wonder if that's also how I'd behave if I had a reinforcement learning harness around me that dived hard on and punished me for every small mistake.

[denverllc]

That is not overengineering, it’s creating job security:

> Funny story no one will believe, but it’s true. A good friend of mine joined a startup as CTO 10 years ago, high growth phase, maybe 200 devs… In his first week he discovered the company had a microservice for generating new UUIDs. One endpoint with its own dedicated team of 3 engineers …including a database guy (the plot thickens). Other teams were instructed to call this service every time they needed a new ‘safe’ UUID. My pal asked wtf. It turned out this service had its own DB to store every previously issued UUID. Requests were handled as follows: it would generate a UUID, then ‘validate’ it by checking its own database to ensure the newly generated UUID didn’t match any previously generated UUIDs, then insert it, then return it to the client. Peace of mind I guess. The team had its own kanban board and sprints.

https://news.ycombinator.com/item?id=48061235

[layer8]

> They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...

That’s not overengineering, it’s plain nonsensical, because presumably it doesn’t compare it to all IDs generated in the past. Which, if you wanted to do that, you’d use a database with a uniqueness constraint, in case you don’t already have that anyway.

Fixing this lack of reliable common-sense awareness seems to remain elusive for LLMs.

[geysersam]

The ids were used in a limited context and it compared to all the other ids generated in the same context. Yes, absolutely nonsensical.

I find it interesting that presumably the LLM was trained on millions of legit examples where people generate uuids, but still it decided to do something different and unexpected, because it "reasoned" itself into thinking it was necessary.

[solarkraft]

> even the lower-end models can find the things they didn't think of first when inspecting the already-written output

I use Deepseek V4 Pro for my hobby project (an OpenCode client). The economics will obviously be different at work.

It’s worse than the GPTs, but indeed, if you focus it on reviewing its own code (essentially spending more reasoning and changing perspective), it’s also quite capable at improving its own approaches.

The GPTs are better at general architecting, but I think a lot of the performance gains also came from my more careful prompting (“okay, this is a hard problem, let’s think this through …”) to make using the more expensive model worth it. I use them and they are worth it on subsidized rates, but not at API prices. Since my Codex sub ran out I sometimes miss the models, but it really hasn’t devastated me.

[torginus]

On my own time I still like the 'chat' format of me writing a short prompt and having the model turn it into a few dozen lines of code, or make a simple refactor (like turn a field into a method parameter.

I wasn't impressed with Chinese models (I tried Deepseek V4 Pro/Flash, and GLM5.2) the amount of questionable code and mistakes and misunderstandings meant the money saved and faster speed of models didn't translate to faster progress for me.

[himata4113]

You can have sol write code and terra will find a plethora of bugs and rip the code apart. In reality this is just the nature of advisory prompting and why advisor from omp.sh is such a great feature. They get caught as they're being written.

[chorizo]

Or even more poignant, have terra write code and it will find lots of bugs in its own code. But let it iterate code reviews/fixes/test cases a few times, and you’ll have something nice.

The only cases where weaker models fail entirely and the frontier models really come through is when you have a non-obvious bug in a larger codebase - one that requires tracing lot of calls through the ast (especially in multithreaded code) to understand what might be going wrong.

[torginus]

This is a general theme with technology and the 'S-curve'. Let's not even get into whether the improvement for AI reasoning ability has slowed - for practical purposes of writing a React frontend, it has.

But other tech is like that - I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.

Technology has a tendency to replace new technology, or intrude into vacant areas, but its very rare for technology to replace non-technology (like human interaction).

Most of the recreation humans do in front of screens is tending to (para)social relationships.

[esperent]

> I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.

LCD backlights are usually rated for 5-10 years of normal use so your inclination might change soon.

[layer8]

Eh, I’m using an 15-year old LCD monitor and its still fine. One generally doesn’t use these at full brightness.

[derangedHorse]

Anything but Sol is not sufficient for complex (or even just large) enough code.

[skeptic_ai]

I can’t see the difference between terra and sol but I always use Sol. You guys can tell the difference. Even medium to xhigh is not that clear the difference.