[zamalek]

> But tests have to be correct.

From my experience, ChatGPT-3 has been pretty good at exercising all meaningful branches (I look at code coverage results, I don't care for percentages at all), in the least amount of tests, on the first go. I definitely have to modify each test quite a bit, because it frequently hallucinates API calls that don't exist, but the code that it produces is an incredible blueprint. And I haven't even attempted to use RCI yet: "improve your answer" or "your answer is wrong because..."; ChatGPT-4 is supposedly extremely adept at reflecting on its responses. I can only imagine where this will be in a few years.

I was about 6 months late to Copilot because I was incredibly skeptic about it, without having used it in anger. My skepticism was mostly (but not entirely) wrong. Having actually used ChatGPT in anger, I find the degree of skepticism extremely skeptical.

It's like picking up C in the 1970s. We're at the very beginning when things are pretty rough, but the skills that I am building today are going to be foundational in the future. If you're dismissing AI without giving it a few weeks to earn its keep, it's going to be rough to catch up when things improve to the point where it is required.

[UncleMeat]

If you write a unit test for each branch in your method and just put in the current behavior as the expectation, all you've done is created a test that says "the method does what it currently does."

[zamalek]

> the method does what it currently does.

It isn't that stupid, and that was done at least a decade ago with static+control flow analysis. As for AI, I was recently writing tests for a VT push parser in Rust (which is novel code, so no parroting here) and it clearly knew enough about VT to write a, correctly, failing test. I had a bug in my parser, and the test that AI generated found it.

At the end of the day, I'm not sure why anyone would believe the critique of someone who hasn't used a tool in earnest.