[NohatCoder]

There is no such thing as a complete test suite, there will always be some possible bug that it doesn't catch.

In particular, if you put an LLM in an automated loop of "this test fails, please fix it", there is a pretty good chance that it will simply special case all of the tests, possibly in some contrived way that makes it not at all obvious when you read the code.

[baranul]

This is what I have observed as well. Been through many debates about the "perfect plan" and "perfect tests" fallacies.

Maybe a way of looking at it, to understand the nature of the issue. Have a LLM translate a novel from English to Spanish. Of course it can do that translation at speeds that no human could (score a point for AI). But how good is the Spanish translation? Is the quality better than what humans could do? Wouldn't those who are not fluent in Spanish be more easily impressed?

We then can do all kinds of configuration setups and tests, but how do we know the Spanish was translated perfectly, without a massive detail review (and being already truly bilingual in both English and Spanish)?

As is the usual case in the pursuit of perfection (which nothing in nature ever seems to be), there is going to be mistakes, costs (worth it?), and gray areas. It would be foolhardy for us not to suspect or pass it off as otherwise.

[deanishe]

> how do we know the Spanish was translated perfectly, without a massive detail review

You can't. I think that's a large part of why LLMs have caught on much better with programmers: they have ways of making the computer check its own work.

Checking a document is still a laborious manual task. And completely unfulfilling.

[baranul]

And that's the trap. Relying on a "perfectly" crafted test or a LLM to verify what is unknown. Yes, LLMs can be very useful, but perhaps it's best for us to be realistic.

> making the computer check its own work

Kind of like the Spanish teacher telling his students they can grade their own tests, then being surprised that Billy was always giving himself 100%, when he's nowhere near that bright or fluent.

It wouldn't be so bad, if people were more upfront with being unsure or made it clear they were extrapolating from smaller and limited data. But usually, like many of these unusually cocky LLMs, what is too often reported to the public is "perfection" and many inconvenient truths "swept underneath the carpet".

[aeonfox]

This is where fuzzing would be useful. We have an at-least-parity-bug-level oracle with the reference PostgreSQL implementation. Just build a generator of queries (both invalid and valid) and ensure the output matches. The yardstick is how many log10(queries) it can go on average before a discrepancy is found.

[apf6]

It’s 2026, so another basic technique that every team should add to their testing strategy (in addition to proven techniques like fuzzing) is agentic user simulation. Set up an AI agent with access to the product, and prompt it to use the thing in hundreds of realistic use cases, and to report any possible bugs. It will catch a lot of ‘blind spot’ bugs that were previously things that were only caught by humans.

[aeonfox]

Debug yourself of the AI hype^. Besides config, SQL implementations have a very limited textual API surface of accepting queries. Putting whole layers of UI and agents around it is really inefficient way of finding edge cases. I wouldn't call that basic, nor the most comprehensive option. Fuzzing at the query layer is far more computationally efficient and effective.

A truly stochastic method is more likely to hit against edge cases, rather than an agent that tends to towards idiomatic solutions and that is trained against a corpus of existing software, and burns millions of tokens/watts spinning its wheels.

^ There's plenty of business value to be found in agentic AI without reaching for it for every solution. I'd even posit agentic AI is even better when paired with focused old-fashioned squishy-brained software engineering in the loop.

[apf6]

you don't have to compare whether stochastic fuzzing or agentic testing is the best. You can just do both of them. They both have enough ROI to be worth the trouble. That's why I said "in addition to".