[maccard]

> You can get the LLM to run a script which checks for all of these

Most of the time. Except for when it forgets to do it.

I think it’s funny that the solution is to use something that is not LLM driven to enforce it.

Also - pre commit hooks aren’t enforced, people will not set them up. You have to run this stuff in CI (which is incredibly annoying given that machines are writing the code in the first place)

[Yokolos]

This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.

[maccard]

I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…

[DCKing]

You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also help agents gain awareness of the overall verification flow including larger things like tests, but that's hard to say for certain.

Putting structural code checks in a precommit hook is arguably better than pulling it into the harness, as it will enforce those constraints no matter whether an agent or human is making the commit.

[maccard]

> , as it will enforce those constraints no matter whether an agent or human is making the commit.

Unless the agent, or the human don't enable the precommit hooks in the first place.

[Muromec]

You don't need the pre commit hooks if you can make the harness hooks.

[QuercusMax]

Belt and suspenders. What if you make a manual code edit? What if your harness hook malfunctions?

[maccard]

What if you don’t enable pre commit hooks?

> what if your harness hook malfunctions?

That’s a bug in the harness and should be enforced

[Muromec]

Writing a custom harness was like 10 bucks of the communist model tokens last month and I ended up with an a actor library, a thing that has a stable identity, writes sutras and knows it wrote the harness itself. Weird.

Adding an auto runner of the unit tests is just... Boring?