[thorum]

Interesting read! Creating tests is highlighted as something Claude did well, but it strikes me that all the weaker rejected solutions could have been avoided if it were really good at designing intelligent tests for itself. For example, the first solution “was very specific to the reported bug and wouldn’t have fixed the general case” and the third suggestion “prevented the perfectly valid use of as conversion expressions in go commands as well”. I imagine both of these cases could have been noticed and avoided by the agent if it had planned out adequate tests ahead of time.

[rapind]

This is kind of what coding with LLMs feels like. Gradually increase guard rails "outside of it's context (automated)" to get the results you want out of it. Static typing, quick compilation, not having nulls, and lints are a great start (I would also argue for managed side effects and functional, but to each their own).

It gets pretty far to the solution on it's own and quickly, but then you spend time adjacent to the problem, building out it's cage while iterating through the remainder of the solution.

[piskov]

As humans we have a concept of viscosity. That resistance, like being in quicksand or a swamp, is how you “easily” identify a code smell, something that needs to be refactored, etc. Part of it is human laziness, part of it some concept of elegance, an itch of being not quite tidy as it can be, etc.

LLM, being a tiresome little helper, will gladly output hundreds of lines, hacks, and what have you.

I don’t think any amount of tests, prompts, harnesses and other “my shaman is a better shaman” will help it to acquire this trait. Some other AI architecture someday maybe — just not today.

And that’s why it is good at what it is and really bad at stuff like code “design” (unless it is a well-known solution being baked in the training set)

[AloysB]

> “my shaman is a better shaman”

This made me chuckle. I will steal this from you.

[nok22kon]

have you tried asking?

I've used with great success prompts like "when implementing this feature, did you encounter sections of code that were needlessly complex, that were making it hard for you to work? what would you change in the design/architecture to make it leaner?"

[piskov]

Everything is just one more prompt away, I swear — literally like a gambling addict with a slot machine.

You forgot the premise of the article and why the proposed solutions were not good. It was not the complexity of a solution: they were simple fast fixes like a tape on a leak, but the hacky tape they were.

(of course I tried, the code after “refactor” is still shit unless you start going very explicit about it at a point of being better and faster of doing it yourself)

[nok22kon]

yes, LLMs are not perfect

this is what separates real engineers which solve hard problems, adapt and overcome, versus the ones which complain that whatever they have access to is not perfect and so they will give up on it because "its unusable"