[ghiculescu]

They are really good if you give them guidance and tight scope. For example for the database migrations review bot, we give it our Cursor rules file on database migrations (which is about 200 lines) and tell it to review the PR based on that.

It works particularly well for migrations because all the context is in the PR. We haven't had as much luck with reviewing general PRs where the reason for a change being good or bad could be outside the diff, and where there aren't as clearly defined rules for what should be avoided.

[eschaton]

How do you square this with the fact that LLM-based tools aren’t actually doing any analysis whatsoever, but are just pachinko machines that produce statistically likely output tokens when given input tokens?