[danpalmer]

PR review sucks because engineers often don't optimise their code authoring for it. When they do it's a joy.

PR review of agent generated code sucks because agents are, in my experience, very poor at authoring for review. This makes sense because the review process is not represented in the resulting code artifact which is what they're trained on. Agents are always producing much larger changes than should be in a single step, and often touch unrelated code with poor judgement of whether it should be included or not.

[preg_match]

One thing that really bothers me about agents is they generate excessively long and detailed comments.

This sounds good for review. Except… the comments are never intent or trade-offs, they’re always the “what”, which the code then duplicates right after. Comments should be short, concise “why”. Often I look at code and think “hm this is odd, I wouldn’t have done it this way, is there some constraint I don’t know about?”

The comment should explain that, not what the code is doing. If I wanted to know what the code is doing I’d read the code!

[latentsea]

Agents are also controllable to a degree. How agents author PRs is now an engineering problem.

[danpalmer]

But this is my point, they're not really controllable in this way. There are no fixed rules (like tests or linters) for small PRs, it's not good enough to do it by line count.

A good PR tells a story to the reader that leaves them confident in the result. That means knowing the reader and how they think, and that is something LLMs are very bad at.

[latentsea]

And you can't think of a way to bring an engineers mindset to solve that?

If you leave it up to raw LLM outputs you're not going to get the results you're after. If you put on your engineering hat and think about your problem statement and what tools you can bring to bear on it you'll find you can likely craft a PR authoring skill and a set of evals for that skill that satisfies your requirements with a high enough degree of reliability.

[danpalmer]

I'm not coming at this cold, I've worked on the skill for PR (CL) creation in the Google monorepo. I'm not talking about raw LLM outputs here. It's a hard problem that is fairly fundamental to the training material being used for LLMs today.

I'm sure we can improve things, I'm sure it will get better over time, but I'd say we're a small fraction of the way towards where we need to be, and trending in the wrong direction with long trajectory development creating bigger changes on average.

[dhorthy]

yes well said

[dhorthy]

exactly. a great pr is a joy to review. we've found some success in agents generating static HTML walkthroughs that order the diffs in something other than GitHub's default alphanumeric ordering, but it can only go so far