[prymitive]

It used to be that you need a good reason to make huge refactorings, because it’s often so much work. Now agent can rewrite half of your code if your prompt is vague enough and you don’t actual try to review it all. And so the “soul” of a program can change dramatically every single day. It’s both great and very much not so.

[Xirdus]

The biggest obstacle to huge refactoring has always been minimizing the risk of bugs, not losing any features, and ensuring compatibility with the existing ecosystem. The reason it's become easier in the age of AI is because we stopped caring about these things.

[sarchertech]

Yep. That’s what people are forgetting. If you have an application that many people depend on to do real work, to make money, you won’t survive if you allow AI to constantly make huge changes.

Your test suite doesn’t cover all workflows. It doesn’t cover every combination of actions a user can take. So every big AI refactor while change some of those.

If this is happening frequently, your software will feel like a janky piece of unusable crap.

[theshrike79]

Actually now we care even more. The reason people didn't write extensive test suites was was because it's super fucking boring to do so.

AIs don't care, they'll happily write 50 unit tests with slight variations and pair them with a full dockerized end to end test suite.

Now we have at least SOME tests. Are they good? Maybe, maybe not - but we have them. If one of them fails later on, we can check if it's an actual issue or the test being bad.

Which is infinitely better than having no tests because nobody can be arsed to write them or the customer isn't paying for the extra hours needed for a proper test suite.

[Xirdus]

People didn't write tests not because it was boring, but because they didn't have time to do it because implementation is more important. That problem didn't go away - it's just instead of not having time to do tests, now we don't have time to review whether the AI wrote tests that make sense. And most of the time it doesn't. They're running the code but not asserting anything. Or they're asserting test inputs instead of implementation outputs. Or outright swallowing errors to make tests pass.

Having such tests is, in fact, worse than not having tests at all. Non-existent tests are just as useful as bad tests, and are much cheaper to maintain.

[zahlman]

Having slop tests instead of no tests is definitely not the same thing as actually caring about avoiding bugs.

[theshrike79]

This is not about caring. This is about the real world where actual human programmers usually don't want to spend any extra time writing test suites or management doesn't give them the time to do so.

So if the starting point is "no tests", having a bunch of "slop tests" is better, right?

[Cthulhu_]

If the "soul" of a program (which is a vague term but I think I get what you mean) changes daily, that's indicative of a lot of churn, and a lot of churn is not a good idea in any project.

it's like changing how the tower of babel should be built daily. Just because you can doesn't mean you should.

[theshrike79]

But if all the tests pass, does it matter what's inside?

[timothycoleman]

Unfortunately the tests can't (usually) cover all behaviours.

And perhaps more importantly, they don't capture more abstract properties of the code like maintainability.

AI works best in well maintained code, but unless care is taken, the AI will (today, anyway) make the code less well maintained as it goes.

If the AI is allowed to introduce mess into the codebase faster than its ability to deal with the mess increases, the codebase will eventually run into a problem that it's hard to recover from.

[theshrike79]

And that's kind of the point, some manual testing is always needed - even when humans write the code.

But you can test things and behaviours to a sufficient degree. Nobody really does 100% test coverage on real world code (outside of medical, airplanes and space of course).

[zahlman]

Yes, of course it does. It's strange to me that this can be asked.

[theshrike79]

Why? How far do you take this, do you inspect the inside of every single thing and program you use?

Your microwave promises to heat your food at 800W for 1 minute, do you open it up and measure the magnetometer strength? Do you double-check the timer that it's accurate?

You want a program that takes in A and returns B within X milliseconds, I deliver it to you. Do you spend the time going through the whole program line by line to see how it was made or do you trust it?

Or when you buy a car do you take it to a garage and dismantle it to see whether it was correctly assembled in the factory? Or do you trust them?

[gart12one]

I think the difference here is that 'passing all the tests' isn't the same as satisfying the contract that the consumer is paying for. Even the best test suites won't cover all of the workflows a user might take.

Being pedantic here, but the two scenarios mentioned have regulations as well with repercussions for those products not satisfying their base requirements, whereas a lot of software is the wild-west (I won't be prosecuted for my software not actually working, in most cases).

[theshrike79]

But what the customer is paying for is defined somewhere, right? Preferably in writing.

You can give that to an agent and have it confirm the application fulfills the spec. Just like a human would have to.

And if the specific test can’t be automated, why? Is it impossible to automate or just inconvenient?

Granted there are immaterial things that we can’t automatically test yet like how the application feels to use. “Snappy” or “smooth” can’t really be quantified with code.

[zahlman]

>You want a program that takes in A and returns B within X milliseconds

No, I actually want a lot more than that. I just didn't communicate all of it, because I'm accustomed to you understanding what good software is. (Or I didn't hire you in the first place, because I don't trust others to care like I do.)

If the microwave doesn't cook my food well, I can probably adjust for it; and anyway the objective power metric isn't the thing that matters to me, but the temperature and taste of the food I pull out. The code is an entirely different story, assuming I'm actually asking for code and not a finished program. Because I'm going to have to integrate it into something else.

And yeah, like the sibling comment said, the currently existing tests aren't actually the full interface. And they, too, are code, which can be wrong.