[zmmmmm]

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so human attention is the primary bottleneck here. Thus poor signal to noise ratio isn't a side issue, it's one of the core issues.

As a result, I'm mostly using this selectively so far, and I wouldn't want it turned on by default for every PR.

[Quarrelsome]

> but the signal to noise ratio is poor

Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a functional issue with the code.

I wonder if AI is ever gonna be able to conquer that one as its quite nuanced. If they do though, then I feel the industry as it is today, is kinda toast for a lot of developers, because outside of agency, this is the one thing we were sorta holding out on being not very automatable.

[eieio]

at my last job code review was done directly in your editor (with tooling to show you diffs as well).

What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss.

This was typically a big shock for new hires who were used to the "comment for every nitpick" system; I think it can feel insulting when someone changes your feature. But I quickly came to love it and can't imagine doing code review any other way now. It's so much faster!

I'm not sure how to tie this to AI code review tbh. Right now I don't think I'd trust a model's taste for when to change things and when to leave a comment. But maybe that'll change. I agree that if you automated away my taste for code it'd put me in a weird spot!

[atq2119]

Was that Jane Street? I remember watching a presentation from someone there about such a system.

If not, any chance this tooling is openly available?

[eieio]

> Was that Jane Street?

yep

[dbtablesorrows]

I think the closest such thing we have is "suggestions" on github and gitlab.

[delusional]

This is the workflow I've always dreamed of. In a lot of ways making a change which is then submitted as patch to their patch isn't really that different from submitting a comment to their patch. The workflow of doing that directly in editor is just wonderful.

If I had to pick, I actually think ONLY being able to submit "counter-patches" would be better than only being able to submit comments. Comments could just be actual programming language style comments submitted as code changes.

[johntash]

What if you have two people with different ideas of how to name a certain variable and they just flip the name back and forth every release?

I like this review method too though, and like that some pr review tools have a 'suggest changes' and 'apply changes' button now too

[pbalau]

> What if you have two people with different ideas of how to name a certain variable and they just flip the name back and forth every release?

Fire both. There is no amount of skill and productivity that can justify that amount of pettiness.

[eieio]

Typically in this system you encode obligations - e.g. "eieio should review, or at least be aware of, all changes made to this library." I think that means you're unlikely in practice to have a problem like that, which (unless the team is not functioning well) requires two people who care deeply about the variable name and don't know that someone else is changing it.

[ljm]

I think it's a good idea to have a style guide of sorts that you can point to when people sweat the small stuff.

[coldtea]

>What if you have two people with different ideas of how to name a certain variable and they just flip the name back and forth every release?

You fire both or at least one of them. Problem solved.

[ta988]

Apply rule 8 of Go

[PurelyApplied]

If minor mistakes are corrected without the PR author's input, do they ever learn to stop making those mistakes themselves? It seems like a system where you just never bother to learn, e.g., style conventions, because reviewers just apply edits as needed.

[Phlebsy]

> What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss.

This is my dream; have only had a team with little enough ego to actually achieve it once for an unfortunately short period of time. If it's something that there's a 99% chance the other person is going to say 'oh yeah, duh' or 'sure, whatever' then it's just wasting both of your time to not just do it.

That said, I've had people get upset over merging their changes for them after a LGTM approval when I also find letting it sit to be a meaningless waste of time.

[simoncos]

Interesting approach. I think it could have the reviewers to be more serious about their feedback. Comments are a bit too casual and may contain more "unconstructive" information.

[vidarh]

I just this morning had someone "nitpick" on a PR I made and ask for a change that would have broken the code.

If the reviewer can make changes without someone reviewing their change, it's just waiting to blow up in. your face.

[eieio]

Yes, in the system I'm describing if a reviewer changed your code, you reviewed their change.

[jcgl]

That sounds great. Was that proprietary tooling? I'd be interested in some such thing.

[eieio]

The tool (iron) isn't open source, but there are a bunch of public talks and blogs about how it works, many of which are linked from the github repo[1].

It used to be "open source" in that some of the code was available, but afaik it wasn't ever possible to actually run it externally because of how tightly it integrated with other internal systems.

[1] https://github.com/janestreet/iron

[gbalduzzi]

If I understood correctly, the same can be done on VS Code with the github plugins (for github PRs)

It's pretty straightforward: you checkout a PR, move around, and either make some edits (that you can commit and push to the feature branch) or add comments.

[jcgl]

Good to know about its existence. I think I'll have to do my own sleuthing though, since I'm a (neo)vim user who dislikes GitHub.

[just6979]

Yeah, it's called git: make your own branch from the PR branch, commit and push the nitpick change, tell the author, and they can cherry-pick it if they approve.

Gitlab has this functionality right in the web UI. Reviewers can suggest changes, and if the PR author approves, a commit is created with the suggested change. One issue with this flow it that's it doesn't run any tests on the change before it's actually in the PR branch, so... Really best for typos and other tiny changes.

Alternatively you actually, you know, _collaborate_ with the PR author, work it out, run tests locally and/or on another pushed branch, and someone then pushes a change directly to the PR.

The complaints about nitpicks slowing things down too much or breaking things sound like solo-hero devs who assume their god-like PRs should be effectively auto-approved because how could their code even contain problems... No wonder they love working with "Dr Flattery the Always Wrong Bot".

*(Hilarious name borrowed from Angela Collier)

[jcgl]

I think you misunderstood the tooling I was asking about. This is what was mentioned:

> at my last job code review was done directly in your editor (with tooling to show you diffs as well).

That's not covered by git itself. And it's not covered by Gitlab, GitHub, or any other web-based forge.

> Alternatively you actually, you know, _collaborate_ with the PR author, work it out, run tests locally and/or on another pushed branch, and someone then pushes a change directly to the PR.

Of course you should collaborate with the author. This tooling is a specific means to do that. You yourself are of course free to not like such tooling for whatever reason.

> The complaints about nitpicks slowing things down too much or breaking things sound like solo-hero devs who assume their god-like PRs should be effectively auto-approved because how could their code even contain problems... No wonder they love working with "Dr Flattery the Always Wrong Bot".

Did you maybe respond to the wrong person? I'm not sure how that relates to my comment at all.

[zenolijo]

Naming comments can be very useful in code that gets read by a lot of people. It can make the process of understanding the code much quicker.

On the other hand, if it's less important code or the renaming is not clearly an improvement it can be quite useless. But I've met some developers who has the opinion of reviews as pointless and just say "this works, just approve it already" which can be very frustrating when it's a codebase with a lot of collaboration.

[daotoad]

Naming comments are useful when someone catches something like:

1. you are violating a previously agreed upon standard for naming things

2. inconsistent naming, eg some places you use "catalog ID" and other places you use "item ID" (using separate words and spaces here because case is irrelevant).

3. the name you chose makes it easy to conflate two or more concepts in your system

4. the name you chose calls into question whether you correctly understood the problem domain you are addressing

I'm sure there are other good naming comments, but this is a reasonable representation of the kinds of things a good comment will address.

However, most naming comments are just bike shedding.

[Arainach]

If the person reading the code doesn't quickly understand what's going on from the name or finds the name confusing, the name is poor and should be changed. It is way too easy for the author to be caught up in their mental model and to be unaware of their implicit assumptions and context and choose a name that doesn't make sense.

The bigger problem is people who feel ownership of shared codebases tied to their ego and who get angry when people suggest changes to names and other bits of interfaces instead of just making the suggested change.

If you get code review feedback, the default answer is "Done" unless you have a strong reason not to. If it's not obvious whether the name suggested by the author or the reader is better, the reader's choice should be taken every time.

[layer8]

> If the person reading the code doesn't quickly understand what's going on from the name or finds the name confusing, the name is poor and should be changed.

I used to think that way, but in many nontrivial circumstances, every conceivable name will be a mismatch for where some person is coming from, and not be self-evident for their mental model. Even the same person, over a longer time span. There is often a gap to bridge from name to meaning, and a comment isn’t the worst way to bridge it.

[hinkley]

I find the thesaurus helps a lot with this. Actually more than just naming, because often a word in the synonym list will stand out as a more accurate representation of the concept you’re trying to add to the code, in a way that reveals subtasks that will substantially increase the value of the feature.

In short I use it as a form of rubber ducking. No it’s not like this word, it’s more like that one, but most of all like this one.

[hinkley]

I’ve seen this enough now to consider it a trope instead of a coincidence. There’s that one or two guys on the team who may be noteworthy in their math clever but only high school reading level, who use the same word in three parts of the code but use a different dictionary definition each time. They don’t see the big deal, they can keep it straight in their head, they insist. And if you can’t then you must be dumb instead of what you really are, which is sick of his bullshit.

Given enough time and rope, these parts of the code start to encroach on each other and the cracks start to show. There are definitely bugs the smart guy introduces because no, in fact, you can’t keep them straight in your head either.

So it does matter if you use, as a top of my head example, the word “account” for both the user and group management features of the app and to describe an entry to an incident report in another part. It will bite you in the ass, and it’s easier to change now when there are three references instead of 23.

[Quarrelsome]

> Naming comments can be very useful in code that gets read by a lot of people. It can make the process of understanding the code much quicker.

yes but it can be severely diminishing returns. Like lets step back a second and ask ourselves if:

var itemCount = items.Count;

vs

var numberOfItems = items.Count;

is ever worth spending the time discussing, versus how much of a soft improvement it makes to the code base. I've literally been in a meeting room with three other senior engineers killing 30 minutes discussing this and I just think that's a complete waste of time. They're not wrong, the latter is clearer, but if you have a PR that improves the repo and you're holding it back because of something like this, then I don't think you have your priorities straight.

[hinkley]

Generally you’d like the variable to imply a call to action. Even if the call to action is for a feature still in the backlog.

Over time I’ve developed some tricks that invite people to add features to the code in the “right” place, and this is one of them. Once in a while someone gets credit for work I already thought to do but didn’t have time. But for every one of those there’s a half dozen or a dozen cases of increasing the bus number on a block of code I wrote be nerd sniping people into making additions while I’m busy with something else.

[spooky_action]

Sorry for the dumb question, is the second version actually better than the first? Because I prefer the first. But perhaps you chose this as a particularly annoying/unuseful comment

[Quarrelsome]

I personally don't give a shit either way but I've worked in dev shops with a clear preference for the second one. I can see their point because the code as natural language parses better but I don't think its strong enough to care about.

Sort of place that is fussy about test naming so where I would do smth like:

TestSearchCriteriaWhere

they'd want

Test_That_Where_Clauses_In_Search_Criteria_Work

I think its a waste of typing but idk, I'm willing to let it slide because I think its a pointless hill to die on.

[tharkun__]

Let's take it up a notch!

    var itemCount = items.Count;

depends on what `items` is, no? Is the `.Count` O(1)? Do you really need a variable or is it fine for the (JIT) compiler to take care of it? Is it O(n) and n is significant enough? Maybe you need a variable and spend time arguing about that name. Yes I chose this because almost everyone I know at least would argue you always have to create the variable (and then argue about the name) ;)

    fussy about test naming

I get fussiness about test naming. I believe that a good test "name" should tell you enough for you to be able to "double check" the test setup as well as the assertions against the test name with some sort of "reasonable" knowledge of the code/problem domain.

As such both of those test names are really bad, because they can't tell anything at all about whether you're testing for the correct thing. How do I know that your assertions are actually asserting that it "works"?

Instead, I'd want a test named something like this (assuming that that's what this particular test is actually about - i.e. imagine this particular test in the context of a user defined search, where one of the options is that they can specify a project to search by and this particular test is about verifying that we check the permissions the user has for said project. There would be different tests for each of the relevant where clauses that specifying a project in the search params would entail and different tests again for each of the other user specifiable parameters that result in one or more where clauses to be generated):

    shouldCheckProjectPermissionsWhenProjectIdInSearchParams()

Every single test case gives you the ability to specify both a good name and clear, concise test assertions. If I see anything but a bunch of assertions related to project permissions for the logged in user in this test, I will fight you tooth and nail on that test ;) I couldn't care less tho if you use camelCase or snake_case or whatever. I just had to choose something to post. I also couldn't care less if you had 17 different assertions in the test (we all know that "rule", right? I think the "test one thing" and "one assertion" is not about the actual number of "assert statements". People that think that, got the rule wrong. It's all about the "thing" the assertions test. If you have 17 assertions that are all relevant to testing the project permission in question then they're great and required to be there. If 1 is for asserting the project permissions and the other 16 are repeating all the other "generic assertions" we copy and pasted from previous tests, then they're not supposed to be there. I will reject such a PR every time.

[solid_fuel]

If I was going to nitpick it I would point out that `itemsCount` could easily be confused with `items.Count`, or vice versa, depending on syntax highlighting. That kind of bug can have a negative impact if one or the other is mutated while the function is running.

So clearly distinguishing the local `numberOfItems` from `items.Count` _could_ be helpful. But I wouldn't ping it in a review.

[layer8]

That’s why it’s `itemCount` and not `itemsCount`. ;)

(Because the correct English term is “item count”, not “items count”.)

Personally, I tend to only name it “count” if it’s a variable that is used to keep a count, i.e. it is continually incremented as new items are processed.

Otherwise I tend to prefer `numItems`.

Yes, this is very close to bike-shedding. There is, however, an argument to be made for consistency in a code base.

[ambicapter]

They’re both equally bad to me, I don’t see the improvement over just using item.count. I may be nitpicking a toy example though.

[Quarrelsome]

I think in this case itemCount had application in a couple of conditions later in the function, so there was value in extracting the count. In my recollection I might be missing some nuance, lets say for the sake of argument it was:

var relevantCount = items.Where(x => x.SomeValue > 5);

vs

var numberOfRelevantItems = items.Where(x => x.SomeValue > 5);

so it wasn't necessarily cheap enough to want to repeat.

[SchemaLoad]

A lot of these comments are not pointing out actual issues, just "That's not how I would have done it" type comments.

[brabel]

And the most amazing part is that we got a mini PR review in the comments to a single line of code someone posted just to show an example of useless debates :D

[xmprt]

Human comments tend to be short and sweet like "nit: rename creatorOfWidgets to widgetFactory". Whereas AI code review comments are long winded not as precise. So even if there are 20 humans comments, I can easily see which are important and which aren't.

[brabel]

We are using BitBucket at work and decided to turn on RovoDev as reviewer. It absolutely doesn’t do that. Few but relevant comments are the norm and when we don’t like something it says we tell it in its instructions file to stop doing that. It has been working great!

[SchemaLoad]

My coworker is so far on this spectrum it's a problem. He writes sentences with half the words missing making it actually difficult to understand what he is trying to suggest.

All of the non critical words in english aren't useless bloat, they remove ambiguity and act as a kind of error correction if something is wrong.

[Quarrelsome]

it "nit" short for nitpick? I think prefixing PR comments with prefixes like that is very helpful for dealing with this problem.

[hdjrudni]

Yes, but I don't know how effective it is. 99% of the time someone leaves a 'nit' the other person fixes it. So we're still dealing with most of them like regular comments. Only once or twice I've been like "nah, I like my way better" but I can only do that if they also leave an LGTM. Sometimes they do. There's one or two people that will hold your code hostage until you reply to every little nit. At that point they don't feel like nits. I always LGTM if the code is functionally correct or if the build breaks in a trivial way (that would also block them from submitting). Then they can address my nits or submit anyway and I'm cool with that.

[wtetzner]

I wonder if there's a psychological benefit though. If someone states up front that they know something is just a nitpick, the author might be less likely to push back, and therefore it's less likely to end up in a bike shedding back-and-forth.

[scott_w]

This and when an author wants to ignore it, they do. You don't need to justify your choice since the person is openly saying "I'm bikeshedding" to you.

[quuxplusone]

> There's one or two people that will hold your code hostage until you reply to every little nit. At that point they don't feel like nits.

If the comment must be addressed before the review is approved, then it is not a nit, it is a blocker (a "changes required"). Blockers should not be marked as nits — nor vice versa.

I agree that prefixing comments with "Nit:" (or vice versa in extreme cases "This is a big one:") is psychologically useful. Yet another reason it's useful is that it's not uncommon for perceived importance to vary over time: you start with "hmm, this could be named blah" and a week later you've convinced yourself it's a blocker — so, force yourself to recognize that it was originally phrased as a nit, and force yourself to come back and say explicitly "I've changed my mind: I think this is important." With or without the "nit/blocker" prefixing pattern, the reviewer may come off as capricious; but with the pattern, he's at least measurably capricious.

[simonbw]

Yes it is. I've really oijed those convention at places I've worked. It probably wouldn't be too hard to instruct AI's to use this format too.

[bentinata]

If you're interested: https://conventionalcomments.org/

It may feels to many. I mostly use suggestion, thought, and todo. When I type down "nit..." I realized it usually does not worth it. I'd rather make comment about higher level of the changes.

[tr_user]

Depends on what you're targeting

- If it's a rough PR, you're looking for feedback on direction rather than nitpicks.

- If it's in a polished state, it's good to nitpick assuming you have a style guide you're intending to adhere to.

Perhaps this can be provided in the system prompt?

[hinkley]

You can do both.

The noise is often what hides the bug in the first place. Aim for more straightforward code and the bug will often surface.

For a while when Node was switching to async from promise chains, people would bring me code or tests that were misfiring and they couldn’t tell why. Often it was because of either a bug in the promise chaining or someone tried to graft async code into the chain and it was an impedance mismatch.

I would start them by asking them to make the function fully async and then come back. About half the time the code just fixed itself. Because the intention of the code was correct, but there was some subtle bookkeeping or concurrency issue that was obfuscated. About a quarter of the time the bug popped out and the dev fixed it. And about a quarter of the time there was a legitimate bug that had been sleeping. The function was fine ones own, but something it called was broken.

There are a lot of situations like that out there. The code distracts from the real problem, but just fixing the “real problem” is a disservice because another real problem will happen later. Make the change easy. That’s always the middle of the solution.

[catlifeonmars]

Let me throw something out there: poor naming obscures and distracts from functional issues. You are right about a good reviewer, but a good author strives for clarity in addition to correctness.

As an aside, naming is highly subjective. Like in writing, you tailor naming to the problem domain and the audience.

[Cthulhu_]

This is why you should set guidelines for reviews (like e.g. https://go.dev/wiki/CodeReviewComments), and ideally automate as much as possible. I'm guilty of this as well, leaving loads of nitpicky code style comments - but granted, this was before Prettier was a thing. In hindsight, I could've spent all that time building a code formatter myself lol.

[renegade-otter]

If you are nitpicking style or conventions that do not have rules in your linting tools, then those should automatically be non-issues, IMO.

[sotix]

I follow a consistent comment pattern[0] that makes blocking vs non-blocking easy to identify.

[0]: https://conventionalcomments.org/

[jolt42]

Don't get me started on not CamelCasing acronyms, acronyms aren't more important than regular words! :)

[causalscience]

Yeah or worse like my boss. We don't have a style guide. But he always wants style changes in every PR, and those style changes are some times contradictory across different PRs.

Eventually I've told him "if your comment does not affect performance or business logic, I'm ignoring it". He finally got the message. The fact that he accepted this tells me that deep down he knew his comments were just bike shedding.

[awesome_dude]

I've been in teams like this - people who are lower on the chain of power get run in circles as they change to appease one, then change to appease another then change to go back to appease the first again.

Then, going through their code, they make excuses about their code not meeting the same standards they demand.

As the other responder recommends, a style guide is ideal, you can even create an unofficial one and point to it when conflicting style requests are made

[causalscience]

> Then, going through their code, they make excuses about their code not meeting the same standards they demand.

Yes!! Exactly. When it comes to my PRs, he once made this snarky comment about him having high expectations in terms of code quality. When it comes to his PRs, he does the things he tells me not to do. In fact, I once sent him a "dis u?" with a link to his own code, as a response to something he told me I shouldn't do. To his credit he didn't make excuses, he responded "I could've done better there, agreed".

In general he's not bad, but his nitpicking is bad. I don't really understand what's going on in his mind that drives this behavior, it's weird.

[zeroCalories]

You should have a style guide, or adopt one. Having uniform code is incredibly valuable as it greatly reduces the cognitive load of reading it. Same reason that Go's verbose "err != nil" works so well.

[brabel]

Style guidelines should be enforced automatically. Leaving that for humans to verify is a recipe for conflict and frustration.

[zeroCalories]

Ideally yes, but there's plenty of cases where that's not desirable or possible.

For example, most people would agree you should use exhaustive checks when possible(matching in rust, records in typescript, etc.). But how do you enforce that? Ban other types of control flow? But even if you find a good balanced way to enforce it, you won't always want to enforce it. There's plenty of good use cases where you explicitly don't want a check to be exhaustive. At which point now you gotta make sure there's an escape mechanism to whatever crackhead check you've setup. Better to just leave a comment with a link to your style guide explaining why this is done. Many experienced developers that are new to rust or typescript simply never think of things like this, so it's worthwhile to document it.

[knes]

At augment code we specifically build our code review tool to find noise to signal ratio problem. In benchmark our comments are 2 to 3x more likely to get fixed compared to bugbot coderabbit etc

You should check it at Augmentcode.com

[marginalia_nu]

That's not even mentioning a not insignificant part of the point of code reviews is to propagate understanding of the evolution of the code base among other team members. The reviewer benefits from the act of reviewing as well.

[materialpoint]

How is that different from today's SA, like CodeQL and SonarQube? Most of the feedback is just sh*t and drives programmers towards making senseless perfections that just double the amount of work had to be done later to toggle or tune behaviour, because the configurable variables are gone due to bad static code analysis. Clearly present intent and convience like: Making a method virtual, adding a public method, not making a method static when it is likely to use instance fields in the future --- these good practices are shunned in all SA just because the rules are opportunistic, not real.

[shakna]

My experience at work: Claude regularly says to use one method over another, because it's "safer"... But the method doesn't actually exist in that language. Seems to get rather confused between C# and C++, despite also getting told the language, before and after getting handed the code.

[greymalik]

It very much depends on the product. In my experience, Copilot has terrible signal noise. But Bugbot is incredible. Very little noise and it consistently finds things the very experienced humans on my team didn’t.

[colechristensen]

One thing I've found to be successful is to

1) give it a number of things to list in order of severity

and

2) tell it to grade how serious of a problem it may be

The human reviewer can then look at the top ten list and what the LLM thinks about its own list for a very low overhead of thinking (i.e. if the LLM thinks its own ideas are dumb a human probably doesn't need to look into them too hard)

It also helps to explicitly call out types of issue (naming, security, performance, correctness, etc)

The human doesn't owe the LLM any amount of time considering, it's just an idea generating tool. Looking through a top ten list formatted as a table can be scanned in 10 seconds in a first pass.

[ljm]

I've only managed to use it as a linter-but-on-steroids because, where I'd normally page through the Ruby docs about enumerators to find the exact method that does what someone has implemented in a PR (because there's almost always something in there that can help out), I can instead prompt to look up a more idiomatic version of the implementation for the ruby version being used. It's easy to cross-check and it saves me some time.

It's not very good with the rest, because there's an intuition that needs to be developed over time that takes all the weirdness into account. The dead code, the tech debt, the stuff that looks fundamentally broken but is depended on because of unintended side effects, etc. The code itself is not enough to explain that, it's not a holistic documentation of the system.

The AI is no different to a person here: something doesn't 'feel' right, you go and fix it, it breaks, so you have to put it back again because it's actually harder than you think to change it.

[jamesfinlayson]

I've been using it a bit lately and at first I was enjoying it, but then it quickly devolved into finding more different minor issues with each minor iteration, including a lovely loop of check against null rather than undefined, check against undefined rather than null etc.

[m463]

> signal to noise ratio is poor

I think this is the problem with just about every tool that examines code.

I've had the same problem with runtime checkers, with static analysis tools, and now ai code reviews.

Might be the nature of the beast.

probably happens with human code reviews too. Lots of style false positives :)

[dakshgupta]

The signal-to-noise ratio problem is unexpectedly difficult.

We wrote about our approach to it some time ago here - https://www.greptile.com/blog/make-llms-shut-up

Much has changed on our approach since then, so we'll probably write a a new blog post.

The tl;dr of what makes it hard is - different people have different ideas of what a nitpick is - it's not a spectrum, the differences are qualitative - LLMs are reluctant to risk downplaying the severity of an issue and therefore are unable to usefully filter out nits. - theory: they are paid by the token and so they say more stuff

[zmmmmm]

very interesting! yes everything you say aligns with my experience and instincts.

[furyofantares]

I agree but find it's fairly easy noise to ignore.

I wouldn't replace human review with LLM-review but it is a good complement that can be run less frequently than human review.

Maybe that's why I find it easy to ignore the noise, I have it to a huge review task after a lot of changes have happened. It'll find 10 or so things, and the top 3 or 4 are likely good ones to look deeper into.

[storystarling]

I suspect the noise is largely an artifact of cost optimization. Most tools restrict the context to just the diff to save on input tokens, rather than traversing the full dependency graph. Without seeing the actual definitions or call sites, the model is forced to speculate on side effects.

[hinkley]

I’d like to see someone try AI guided property based testing. The random walk could take a long time ti notice problems with code. I’d trust AI more if it brought proof instead of speculation

[iryna_kondr]

My experience is similar. AI's context is limited to the codebase. It has limited or no understanding of the broader architecture or business constraints, which adds to the noise and makes it harder to surface the issues that actually matter.

[matsemann]

It also acts as mainly an advanced linter. The other day it pointed out some overall changes in a piece of code, but didn't catch that the whole thing was useless and could've been replaced with an "on conflict to update" in postgres.

Now, that could happen with a human reviewer as well. But it didn't catch the context of the change.

[thesurlydev]

For the signal to noise reason, I start with Claude Code reviewing a PR. Then I selectively choose what I want to bubble up to the actual review. Often times, there's additional context not available to the model or it's just nit picky.

[just6979]

Wait, so you have the LLM review, then you review (selectively choose) the proposed review, then you (or the LLM?) review the reviewed review? But often times (so, the majority?) the initial LLM review is useless, so you're reviewing reviews that won't pass review...

Sounds incredibly pointless. But at least you're spending those tokens your boss was forced to buy so the board can tell the investors that they've jumped on the bandwagon, hooray!

[lanyard-textile]

Agreed.

I have to constantly push back against it proposing C++ library code, like std::variant, when C-style basics are working great.

[CSMastermind]

You should try Codex. There's a pretty wide gap between the quality of code review tools out there.

[biophysboy]

I absolutely hate the verbosity of AI. I know that you can give it context; I have done it, and it helps a little. It will still give me 10 "ideas", many of which are closely related to each other.