Au contraire - LLMs are quite bad at large scale pattern fidelity. They'll even forget key details and constraints unless told over and over again. That's why AI-written code has the quality of a patch-on-patch-on-patch.
Fully agree. I tried to refactor parts of a large code base with Fable+ultracode and it just keeps accidentally merging distinct concepts and making up explanations/reasonings that the code base did not contain.
For example, the code base contains a physical controller. It’s closed loop in that it can react in realtime to changes. But it’s a slightly untypical implementation because this one can even look into the future through simulations. But Fable does not understand that. Instead, I need to remind it every 30 minutes that this is closed loop. It keeps wrongly claiming that the controller was open loop and then based upon that it will make up constraints that don’t actually exist.
I feel like there are a lot "you are holding it wrong" arguments flying around. Like, when somebody says that AI wasn't able to accomplish something, people tend to assume it's an User problem.
Meanwhile, I have a hard time to believe people don't encounter problems with AI solutions on a regular basis (I do).
mostly the problem with coding is the semantic ambiguation; coders like to reuse similar methods, variable names, copy/paste etc; so large code bases have so much out of context simularities that the LLM, regardless of size, isn't
The patch-on-patch-on-patch is exactly right, nice way to describe it. It feels like, and I think is, optimized to find the quickest answer not necessarily the right answer.
"It is not LLMs fault but you not knowing how to write a prompt". I know I know. But just giving all codebase and saying "rewrite it" is a no go. If e.g. going one class after class LLM will be exceptionally good at keeping the patterns and logics.
I mean it is a tool and you need to understand how the tool works. When there is too little context, where there is so much context so that you are poisoning it, when you are allowing the tool to do patch-on-patch and etc.
That’s why junior devs generally aren’t give the responsibility of architecting a large scale refactor. Yet people seem to be trying to had these types of tasks over to LLMs.
I’ve been hearing for quite some time now that I should be using an LLM to plan before the build. This is treating the LLM as the architect, not as a junior being handed small tasks here and there.
I haven’t bought into it, so don’t use it this way, but an army of people online and in the media are pushing the fomo hard and telling people this is how it should be used. If the LLM isn’t doing what we want, it simply means we need to use the LLM even more. That’s the prevailing message from the industry.
For example, the code base contains a physical controller. It’s closed loop in that it can react in realtime to changes. But it’s a slightly untypical implementation because this one can even look into the future through simulations. But Fable does not understand that. Instead, I need to remind it every 30 minutes that this is closed loop. It keeps wrongly claiming that the controller was open loop and then based upon that it will make up constraints that don’t actually exist.