Hacker News new | ask | show | jobs
by doodlesdev 1141 days ago
That's really interesting, indeed I can reproduce this by changing the comment. I also managed to get correct output for this sample by renaming the function.
1 comments

clearly your original comment was unfair.
Is it, though? The major selling point of coding LLMs is that you can use natural language to describe what you want. If minor changes to wording - the ones that would not make any difference with a human - can result in drastically worse results, that feels problematic for real-world scenarios.
The model is small, so it has weaker semantics.
I get that. But they are explicitly comparing it to Codex themselves.
The criticism stands if you have to continue to rewrite your "prompt" until you can coax out the correct desired output.