[OtomotO]

Lol. If you're doing anything non trivial that's not a CRUD webapp but e.g. some physics simulation or high performance GPU code any and all models I've tried suck.

They are not just leagues behind what experts would code, they are not even playing the same game.

Which is to be expected, as there isn't so much physics or high performance gpu code available as there is for your typical CRUD API and JS frontend.

[rweichler]

I can attest to this, I had a very simple 20-line shader that I asked Claude to do a basic 90-degree rotation on it, and it just completely got it wrong. Frequently adds pointless abstractions / intermediate variables even when I tell it explicitly not to in the system prompt. I can go on and on, these things just don't understand architecture. And why would they? They were trained on text.

There is something remarkable about turning speech into code (don't need to hunch over a keyboard nearly as much these days, can just talk into a mic) and it's good for first drafts / exploring ideas. But it's obvious to anyone that's paying attention we're hitting the top of the S-curve. It's no wonder the IPOs are around the corner. I mean even Dario admitted he doesn't know how they're gonna substantially increase the context window size. That says a lot.

[rweichler]

That being said I think the harnesses are only getting better. And maybe we will get multi-modal models that understand architecture eventually. But the growing-the-blob-of-text training method that's being used now appears to be getting diminishing returns