[i_have_an_idea]

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs just getting it right at the 1st/2nd attempt.

[hannofcart]

Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.

You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding.

I wrote down a more detailed post of the various linters I use here:

https://www.balajeerc.info/Use-Deterministic-Guardrails-for-...

[rafaelmn]

> Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.

I have legacy endpoints that are no longer used in practice, there for historical reasons, intertwined with existing code etc. They might be marked obsolete, services implementing it are not - agent greps those, builds off of them - produces half legacy garbage.

Linters only handle trivial cases most of us already solved.

[yoyohello13]

Yeah, we have a big struggle with this. We have lots of legacy code that doesn't follow our latest design patterns intermixed with new code. The LLM picks up bad habits depending on what it pulls in to context first. We have AGENTS.md configured with the right way, but old style still slips in. We obviously need to update the old code but on the other hand if it ain't broke why touch it.

[kstenerud]

I have the agent inject comments that mention that this particular code is legacy and must not be used as a reference, should not be cleaned up, etc. If you have a document that lists all of the reasons not to use or touch some code, the comments can simply be references to it.

    // LEGACY CODE, per docs/legacy_rules.md §14, §19

[yoyohello13]

Thanks for the tip. I’ll try that out.

[saghm]

Sure, giving agents a deterministic way to run linters is good, but that already implicitly accepts the premise that code hygiene does actually matter. If you agree with that, then you probably also recognize that there are some things that you can't lint for because they're a lot harder to define, and why would you expect an agent to be able to figure that out today when they still need our intervention to even properly run linters?

[maccard]

> You can get the LLM to run a script which checks for all of these

Most of the time. Except for when it forgets to do it.

I think it’s funny that the solution is to use something that is not LLM driven to enforce it.

Also - pre commit hooks aren’t enforced, people will not set them up. You have to run this stuff in CI (which is incredibly annoying given that machines are writing the code in the first place)

[Yokolos]

This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.

[maccard]

I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…

[DCKing]

You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also help agents gain awareness of the overall verification flow including larger things like tests, but that's hard to say for certain.

Putting structural code checks in a precommit hook is arguably better than pulling it into the harness, as it will enforce those constraints no matter whether an agent or human is making the commit.

[maccard]

> , as it will enforce those constraints no matter whether an agent or human is making the commit.

Unless the agent, or the human don't enable the precommit hooks in the first place.

[Muromec]

You don't need the pre commit hooks if you can make the harness hooks.

[Muromec]

Writing a custom harness was like 10 bucks of the communist model tokens last month and I ended up with an a actor library, a thing that has a stable identity, writes sutras and knows it wrote the harness itself. Weird.

Adding an auto runner of the unit tests is just... Boring?

[gitaarik]

Why not just fix the code before you make a new feature? Or is that what you mean? Only you automatically fix the code on every prompt?

[_the_inflator]

If you have a large important financial service to maintain and develop, and such a code basis is part of your backend, I guarantee you that you won't think twice to simply rely on deterministic linters.

The reason is quite obvious if you have dealt with such a huge code basis in production with thousands of developers contributing for decades to it coming from different vendors and countries.

Code has a meaning attached to it. And paradoxically being able to cleanly cut out such dead code raises my suspicion. There is a reason why such code exists in there often times and since almost always stakeholders give a damn about documentation, and developers traditionally have a hard time writing even JavaDocs, JSdocs, whatever and not to mention maintaining them.

In earlier times CPU time was precious and comments were deliberately left off due to space and processing considerations.

So why is this all important?

Because until you cannot find the one guy who uses this code for a good reason, I would never kill it. Good reasons in these cases are almost always so called application owner, an app admin and hosts, who serves according to ITIL specs as deployment and production person.

Dead code can be actually quite lively under the right circumstances. And since sometimes people have to be very creative to serve regulation requirements and compliance, sometimes release pressure or missing tools can make such code an important script or deployment tool or fix for a reboot or whatever.

Believe me, dead code isn't. What you can do is, watch it at least over a period of two years.

Here is why.

Most processes have yearly deadlines. Many fall on the 1.1. of each year, while others somewhere at the end of the year. Some processes need to be served once a year due to compliance to laws.

And why two years then? As I said, human beings. Maybe that one time a guy had an exception running for it or there is a maintainer, who uses a different method - his own dead code so to say - because these scripts are rarely shared and maintainer's best kept secrets. The lesser a company knows, the more important these last line folks feel and they are blackboxes when something is or isn't working. (I hated this, this was not way of working and I changed it. It is not their company and a keeper is a Red Flag for me.)

But rarely the same person will serve the process two times in a row. Vacation times vary as well as positional changes.

Hence the two years period and even then, there are smarter, more easier ways than to use linters.

Hint: it is the frontend first paradigm.

How do I know? Because I invented it. Proof: Huge international Bank. dbCORE.

[yoyohello13]

I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.

[jaggederest]

It actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing, but the context for agents is taking up the same space and "attention" every time they're run, and they're creations entirely of context, so the quality and examples matter massively.

[maccard]

> actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing,

You’ve never had an agent completely lose the plot and forget/confuse its instructions due to the context filling up?

[Muromec]

That's the moment it should take a nap to compress the context. If you are expose the context size to the agent through some diagnostic input it will even do it itself

[embedding-shape]

> I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one.

I guess it depends on what you mean with "better" but almost all the agent-built projects I do with zero regards to code quality, design and architecture ends up with every single agent needing 10+ minutes to do even the easy changes, while the ones where I focused on those things together with the agent, large changes can take 10+ minutes but everything else is solved faster.

I don't have empirical evidence of this yet, I guess I should put together some sort of test to confirm/disconfirm this.

[BobbyTables2]

Feel the same way myself when working in messy codebases… At some point, the horrible patterns start to rub off…

[NitpickLawyer]

Every time this subject comes up, there are a bunch of takes along the lines of "would you work on a codebase maintained by agents? they'll mess up the code". And I'm asking myself where these people work, because in 20+ years I've yet to see that pristine state of a project that keeps being pristine after the honeymoon greenfield phase, and 50+ people start working on it. Every project devolves in time, old stuff gets patched in a hurry, someone tries to make it better, learns why certain things were done a certain way, hits some undocumented client needs handled by some arcane combination of code + external systems, and so on. If anything, keeping track of what does what in a project is a task where agents can shine, if only in "ask" mode so you can figure out things quicker. Not to mention onboarding and stuff for new team members.

[Muromec]

Entropy is real and with offshore contractors you don't need agents to blame for it

[ramraj07]

I was reading your comment, agreeing with it but still feeling why this is a bad comment. It just occurred to me that an anecdotal statement like this is the antithesis of scientific discourse. We have a paper here, trying to answer a question, and anecdotal testimonials can only harm the discussion by biasing readers without adding anything of value to let anyone objectively conclude anything on the problem.

The most useful discussion would be if we all read the paper and critique its methodology or results.

[vetronauta]

I was reading your comment, disagreeing with it but still feeling why this is a good comment. It just occurred to me that this is not science: science must be reproducible and this is just an historical report on artifact that will be unavailable soon.

[dnautics]

i mean this is feeling too but im too paranoid and frequently do refactoring and code organization passes and never don't do it, so i cant say i know for sure there is a delta.

though people who complain that llms aren't that great strike me as the type to have messy code bases