[SillyUsername]

o3 has been the worst model of the new 3 for me.

Ask it to create a Typescript server side hello world.

It produces a JS example.

Telling it that's incorrect (but no more detail) results in it iterating all sorts of mistakes.

In 20 iterations it never once asked me what was incorrect.

In contrast, o4-mini asked me after 5, o4-mini-high asked me after 1, but narrowed the question to "is it incorrect due to choice of runtime?" rather than "what's incorrect?"

I told it to "ask the right question" based on my statement ("it is incorrect") and it correctly asked "what is wrong with it?" before I pointed out no Typescript types.

This is the critical thinking we need not just reasoning (incorrectly).

[echoangle]

> Ask it to create a Typescript server side hello world. It produces a JS example.

Well TS is a strict superset of JS so it’s technically correct (which is the best kind of correct) to produce JS when asked for a TS version. So you’re the one that’s wrong.

[pton_xd]

> Well TS is a strict superset of JS so it’s technically correct (which is the best kind of correct) to produce JS when asked for a TS version. So you’re the one that’s wrong.

Try that one at your next standup and see how it goes over with the team

[redox99]

He's not wrong. If the model doesn't give you what you want, it's a worthless model. If the model is like the genie from the lamp, and gives you a shitty but technically correct answer, it's really bad.

[echoangle]

> If the model doesn't give you what you want, it's a worthless model.

Yeah, if you’re into playing stupid mind games while not even being right.

If you stick to just voicing your needs, it’s fine. And I don’t think the TS/JS story shows a lack of reasoning that would be relevant for other use cases.

[hatefulmoron]

> Yeah, if you’re into playing stupid mind games while not even being right.

If I ask questions outside of the things I already know about (probably pretty common, right?), it's not playing mind games. It's only a 'gotcha' question with the added context, otherwise it's just someone asking a question and getting back a Monkey's Paw answer: "aha! See, it's technically a subset of TS.."

You might as well give it equal credit for code that doesn't compile correctly, since the author didn't explicitly ask.

[SillyUsername]

As I mentioned TS/JS was only one issue (semantic vs technical definition), the other is that it didn't know to question me, making it's reasoning a waste of time. I could have asked something else ambiguous based the on context, not a TS/JS example, it likely would still not have questioned me. In contrast if you question a fact, not a solution, I find LLMs are more accurate and will attempt to take you down a notch if you try to prove the fact wrong.

[SillyUsername]

Well yes, but still the name should give it away and you'll be shot during PRs if you submit JS as TS :D

The fact is the training data has confused JS with TS so the LLM can't "get its head" around the semantic, not technical difference.

Also the secondary point wasn't just that it was "incorrect" it's the fact its reasoning was worthless unless it knew who to ask and the right questions to ask.

If somebody says to you something you know is right, is actually wrong, the first thing you ask them is "why do you think that?" not "maybe I should think of this from a new angle, without evidence of what is wrong".

It illustrates lack of critical thinking, and also shows you missed the point of the question. :D