The biggest challenge i found with LLMs on large codebase is making the same mistakes again and again How do keep track of the architecture decisions in context of every tasks on the large codebase ?
Very very clear, unambiguous, prompts and agent rules. Use strong language like "must" and "critical" and "never" etc. I would also try working on smaller sections of a large codebase at a time too if things are too inaccurate.
The AI coding tools are going to be looking at other files in the project to help with context. Ambiguity is the death of AI effectiveness. You have to keep things clear and so that may require addressing smaller sections at a time. Unless you can really configure the tools in ways to isolate things.
This is why I like tools that have a lot of control and are transparent. If you ask a tool what the full system and user prompt is and it doesn't tell you? Run away from that tool as fast as you can.
You need to have introspections here. You have to be able to see what causes a behavior you don't want and be able to correct it. Any tool that takes that away from you is one that won't work.
> Use strong language like "must" and "critical" and "never" etc.
Truly we live in the stupidest timeline. Imagine if you had a domestic robot but when you asked it make you breakfast you had to preface your request with “it’s critical that you don’t kill me.”
Or when you asked it to do the laundry you had to remember to tell it that it “must not make its own doors by knocking holes in the wall” and hope that it listens.
There is even a chance our timeline might include that robot too at some point...
Book recommendation no one asked for but which is essentially about some guy living through multiple more or less stupid timelines: Count to Eschaton series by John C. Wright
I start my sessions with something like `!cat ./docs/*` and I can start asking questions. Make sure you regularly ask it to point out any inconsistencies or ambiguity in the docs.
In some sense “the same mistakes again and again” is either a prompting problem or a “you” problem insofar as your expectations differ from the machine overlords.
The AI coding tools are going to be looking at other files in the project to help with context. Ambiguity is the death of AI effectiveness. You have to keep things clear and so that may require addressing smaller sections at a time. Unless you can really configure the tools in ways to isolate things.
This is why I like tools that have a lot of control and are transparent. If you ask a tool what the full system and user prompt is and it doesn't tell you? Run away from that tool as fast as you can.
You need to have introspections here. You have to be able to see what causes a behavior you don't want and be able to correct it. Any tool that takes that away from you is one that won't work.