[facundo_olano]

Author here. I'm surprised to see this surfacing now. I just wanted to clarify, since apparently the post doesn't do a good job at it, that what I discussed there is not a methodology I advocate for. The point of the post was: ok, since there are organizations mandating to maximize speed by reducing time spent on typing code (or even mandating to maximize agents usage), is there a way we can meet that requirement while still preserving the rigor somewhere else?

This was a follow up to a previous article[1] and the pair tried to express what I still think today (using AI daily at work): every time I use AI for coding, to some capacity I'm sacrificing system understanding and stability in favor of programming speed. This is not necessarily always a bad tradeoff, but I think it's important to constantly remind ourselves we are making it.

[1] https://olano.dev/blog/tactical-tornado/

[jimmaswell]

> every time I use AI for coding, to some capacity I'm sacrificing system understanding and stability in favor of programming speed.

Sure, but couldn't you say the same for letting other people contribute code too? In either case, you make the choice of how deeply you want to review it. You can ask the AI or the human to explain things that aren't clear.

For me it's case by case in either scenario. Sometimes it's not that important to look closely at a specific subsystem that's self-contained or just simple, other times I need to carefully audit whatever touches a different system. You need a good sense of the existing codebase/architecture in the first place to make these determinations.

[kenhwang]

I always wondered why people don't also ask the AI to generate code comments/documentation, summaries of those documentation, overview of the system, and re-review them all for correctness for the changes they asked the AI to do.

What I've noticed reviewing all my colleagues' AI generated code PRs is: it really is just code, and the rare comment here and there is still added by the human.

We're already trying to light tokens on fire as fast as possible to stay on acceptable required use leaderboards, why not light some more for system understanding and housekeeping.

[ossopite]

The docs/summaries part I can get behind if reviewed and improved by a human, but at least when working in pre-existing codebases, I tend to steer models away from writing comments, because I find that almost all comments they write are "not even wrong".

That is: they either reiterate what the code does, or would if the code were slightly clearer, or they tell half truths that are more confusing than helpful. Mostly they fail to emphasise the salient things, like the why over the what, that are not obvious from the code.

[steinnes]

100% agree. 95% of the AI comments I see are redundant, misleading or both. Commonly they feel like the reasoning process leaking into code.

[apsurd]

yes it makes sense that it's AI all the way down, but also the "why not just…" answer is because it's exhausting.

Being reduced to an inconsequential middle manager is more exhausting than being reduced to a code monkey is the hot insight i've been hanging my hat on.

To be clear: This is really horrible in an IC for the paycheck role. I quit my job on principle because of code/token maxing. Very few are in the place to do this. I've been enjoying AI as an independent, but i still mean to fight the good fight for every line engineer.

[khasan222]

> I always wondered why people don't also ask the AI to generate code comments/documentation, summaries of those documentation, overview of the system, and re-review them all for correctness for the changes they asked the AI to do.

I now on all of my projects have an ai journal that stands as a ledger for every change the ai has made, and why it was made. I don’t read it that hard personally because I spend so much time planning with my agent before letting it code. However I have found it very useful in sharing code between people, or having Claude look through the journal to gain context when modifying or adding a feature.

[cpeterso]

I see people asking LLMs to omit comments, but comments like PDL (Program Design Language) could be helpful for interactive development: ask the LLM to write pseudocode comments, review them, and then ask the LLM to expand them to actual code. People say comments should explain why, not how. That seems useful for both human reviewers and LLMs.

https://en.wikipedia.org/wiki/Program_Design_Language

[StilesCrisis]

What harness? In my experience, Claude shoots out tons of very low-quality comments. It's always too hyper-focused on the exact specifics of the bug as presented to it, with no higher-level generalization of the concepts involved. In a big codebase, this means the comments are meaningless without a human rewrite, but it definitely writes them.

[alchemism]

I’ve have amazing, amazing results through composing an “ADR” skill hook. It knows contextually what they are for; how to write them; when to include them as steering.

[stuaxo]

You get LLMs like Gemini that will assure you they are right.

There seems to be a big push for 0 reviewing, which is insane.

[distances]

The review needs to be done with clear context, or another LLM altogether.

[weird-eye-issue]

For major new features, I do ask it to do those things. Why is it that you think people don't do that?

Even for most small changes I will ask it to do a simple "production review" then I use my experience and judgment to decide on which items need to actually be addressed or not

[stuaxo]

The people can meaningfully collaborate and produce something of high quality.

[jimmaswell]

I see you haven't met some of my coworkers.

[khasan222]

For sure every time you use ai you’re sacrificing understanding if you don’t plan out and understand how exactly the ai is going to do the work you asked it to do.

The same output that is such a bad thing in this article can also be used to gain context, by making a thorough plan with your ai first, reading through the plan and proposing changes just like you would with a real developer.

You can also use this output to have the ai write a journal as well. The journal can be as detailed as possible and essentially a ledger of all of the changes your ai has made to the code. This allows not only for your teammates reviewing your pr to gain greater context, but also can be used by yourself, or even the ai itself to figure the why behind a particular implementation was done the way it was, far into the future even.

Lastly how many of us ever deploy code without actually checking the feature works e2e? I would gather not many of us do, I don’t, because even though we may have a greater understanding of the code, we can make mistakes in the code or in our logic. And I keep coming back to why would we treat llms any differently? I believe we should be spending our energy thoroughly manually testing a feature to make sure when we brainstormed we actually did get every edge case, and it works well.

[sokoloff]

I think most people test at least a happy path of their code end to end. I think we can all agree that your last sentence is far more aspirational than bare minimum standard practice. (“I believe we should be spending our energy thoroughly manually testing a feature to make sure when we brainstormed we actually did get every edge case, and it works well.”)

I did one small side web project by only writing spec tests and prompts and testing the results in a browser, never reading nor editing a single line of generated code. It was something for home and so low stakes, but it worked remarkably well and was much better tested than the typical 2022-era home project of mine.

[khasan222]

My last sentence is definitely aspirational, it is how I try and go about it, but for sure I make mistakes. However your comments about writing spec tests was interesting to me.

Honestly I don’t even write tests manually because of coverage checks. Being that the coverage check is not something easily manipulated, I always tell the ai, don’t ever change configs, and make the coverage pass whatever I set it to, most times > 95%. I just tell the AI, make this coverage pass.

I find tremendous success with this technique, or anytime really I can find an objective way for the ai to test its work.

[gcr]

What does “much better tested” mean to you?

If you don’t read the tests to check they confer your intent or specifications, they’re more like tautologies than tests, you know?

[sokoloff]

I don't understand the comment.

I wrote the tests. That was how I expressed the spec and my intentions.

[gcr]

Ah. Most Claude users let Claude write their tests for them and I assumed you were too. Sorry.

[ignoreusernames]

Don’t you think that the provider of the LLM is also a dimension on these discussions about responsibility? We often talk about the tech itself (LLM driven development) but how we access it is just as important imo. It’s either locked behind a non trivial amount of hardware (for open models) or some monopolistic driven provider entity like OpenAI or anthropic. In the provider case, it’s not really the LLM that will “own” the code, it’s the provider itself and we’ll be at the mercy of whatever pricing model they shove down our throats.

[LelouBil]

I don't like the premise of the article, but I agree that if you accept the premise, the contents of the articles are a good way to do it.

[dapperdrake]

It gets better:

(NOT a lawyer)

Previously, liability and indemnification could be bureaucratically laundered to "engineers", because it was a huge diffuse set of people.

Now the bag is left with top of the chain for authorizing LLMs. Gia Tan went the hard way with xz. LLM-trolling is the new social engineering.