[jasonthorsness]

There's a lot of precedence for excluding "generated" code from review and linting but it's a bit weird when that is a large portion of the application and you can't rely on the correctness (non-LLM generated code is typically very mechanically generated so you just need to ensure the inputs are correct, not outputs).

I think testing and reviewing LLM-generated code remains just as important. Hopefully they will get better, and it will be easier (and hopefully LLMs can also assist with reviews).

[jollyllama]

Indeed, it is bizarre to see the this approach emerging, in which non-deterministically generated code is treated the same as deterministically-generated. They're two categorically different things and it's as though everyone forgot all of those LLM primers from two years ago that explained why LLM output is non-deterministic and what that means.

[knome]

yeah, traditionally 'generated code' is really just 'decompressed code'. it's got some source of information that's either brought in or crafted to be easily read and modified by people, and a transformer to expand it in a regular way that means, assuming the transformer isn't flawed, the code will always be correct.

if there was an `if( randomly() ){ emit("error"); }` in there, we'd be right back to reviewing it and probably wouldn't bother with it in the first place. besides which, any work to the transformer itself necessitates review even for generated code, making sure that the output is actually what you expect it to be.

the idea that you shouldn't care what's in a function because your possibly-insufficient-over-the-interface tests passed is kind of insane.