[screye]

How do you maintain high confidence in the code it generates ?

My current bottleneck is having to review the huge amounts of code that these models spit out. I do TDD, use auto-linting and type-checking.... but the model makes insidious changes that are only visible on deep inspection.

[sneak]

You have to review your code for quality and bugs and errors now just as you did last month or last year. Did you never write bugs accidentally before?

We're all bottlenecked on reviewing now. That's a good thing.

[screye]

There was a greater awareness of exactly what I'd written. By definition, I would not have written those bugs in, as long as I had known edge cases in my mind.

Lapses of judgement and syntax errors happen, but they're easier to spot because you know exactly what you're looking at. When code is written by a model, I have to review it 3 times.

1st to understand the code. 2nd to identify lapses in suspicious areas. 3rd to confirm my suspicions through interactive tests, because the model can use patterns I'm unfamiliar with, and it takes me some googling to confirm if certain patterns used by the model are outright bugs or not. The biggest time sink is fixing an identified bug, because now you're doing it in someone-else's (model's) legacy code rather than a greenfield feature implementation.

It's a big productivity bump. But, if reviewing is the bottleneck, then that upper bounds the productivity gains at ~4x for me. Still incredible technology, but the death of software-engineering that it is claimed to be.