[Keyframe]

I don't want to knock you down as most have already did. In-fact it's a useful exercise going forward in exploring how to work with AI. It's here, we're all going to use it one way or the other. Zero issues with that, in-fact kudos to going through the pain of it all.

Now, having gone through several such endeavors originally myself, albeit with internal tools and systems (as an exercise), I've noticed that while all my tests passed with flying colors the rewrite itself was broken even on basic functionality or missed a ton of details. It was in effect useless when I dived into it. Initial tests also showed massive gain in performance, and I know people who were involved aren't really dumb so something smelled funny. Turns out all those things left out and honestly... moments were the key ingredients.

What I did learn from those beginning explorations though was that one-shotting, grand architecture or source up-front, master plans up-front.. all these do not yield good results - YET. Who know what we'll see in few years though. What I did found that works (FOR ME, nota bene) is to keep the design and checklists for myself, written by myself and then do a small piece by piece.. as if you would if you were coding alone or if you would waterfalling a small team of talented juniors. Then, suddenly super happy results come out, but then it's mostly you driving all the way where llm writes code and offers advice (which for the most part you ignore). It's a happy place for myself at least. It's then truly unlocking yourself to the mythical 10x.

Rewriting a large proven system with decades of ultra expertise behind it, which I don't have, is guaranteed not to end up the same 1:1 replacement. If you found a recipe for that - please do share.

[gulugawa]

I don't code with LLMs, and think you might be right.

However, Postgres is a tool with clearly defined functionality and doesn't have ambiguous requirements that are seen in user facing software. As a result, it is entirely plausible that the author created a working Postgres replacement for certain use cases.

I personally want to see more evidence about the quality of the tool after it is in a finished state.

[ramijames]

I'm curious. Do you attribute this to weak and/or incomplete tests? How granular should tests be to have complete coverage so that an AI won't create a converted codebase that "passes tests" but is still functionally inaccurate?

[NohatCoder]

There is no such thing as a complete test suite, there will always be some possible bug that it doesn't catch.

In particular, if you put an LLM in an automated loop of "this test fails, please fix it", there is a pretty good chance that it will simply special case all of the tests, possibly in some contrived way that makes it not at all obvious when you read the code.

[baranul]

This is what I have observed as well. Been through many debates about the "perfect plan" and "perfect tests" fallacies.

Maybe a way of looking at it, to understand the nature of the issue. Have a LLM translate a novel from English to Spanish. Of course it can do that translation at speeds that no human could (score a point for AI). But how good is the Spanish translation? Is the quality better than what humans could do? Wouldn't those who are not fluent in Spanish be more easily impressed?

We then can do all kinds of configuration setups and tests, but how do we know the Spanish was translated perfectly, without a massive detail review (and being already truly bilingual in both English and Spanish)?

As is the usual case in the pursuit of perfection (which nothing in nature ever seems to be), there is going to be mistakes, costs (worth it?), and gray areas. It would be foolhardy for us not to suspect or pass it off as otherwise.

[deanishe]

> how do we know the Spanish was translated perfectly, without a massive detail review

You can't. I think that's a large part of why LLMs have caught on much better with programmers: they have ways of making the computer check its own work.

Checking a document is still a laborious manual task. And completely unfulfilling.

[baranul]

And that's the trap. Relying on a "perfectly" crafted test or a LLM to verify what is unknown. Yes, LLMs can be very useful, but perhaps it's best for us to be realistic.

> making the computer check its own work

Kind of like the Spanish teacher telling his students they can grade their own tests, then being surprised that Billy was always giving himself 100%, when he's nowhere near that bright or fluent.

It wouldn't be so bad, if people were more upfront with being unsure or made it clear they were extrapolating from smaller and limited data. But usually, like many of these unusually cocky LLMs, what is too often reported to the public is "perfection" and many inconvenient truths "swept underneath the carpet".

[aeonfox]

This is where fuzzing would be useful. We have an at-least-parity-bug-level oracle with the reference PostgreSQL implementation. Just build a generator of queries (both invalid and valid) and ensure the output matches. The yardstick is how many log10(queries) it can go on average before a discrepancy is found.

[apf6]

It’s 2026, so another basic technique that every team should add to their testing strategy (in addition to proven techniques like fuzzing) is agentic user simulation. Set up an AI agent with access to the product, and prompt it to use the thing in hundreds of realistic use cases, and to report any possible bugs. It will catch a lot of ‘blind spot’ bugs that were previously things that were only caught by humans.

[aeonfox]

Debug yourself of the AI hype^. Besides config, SQL implementations have a very limited textual API surface of accepting queries. Putting whole layers of UI and agents around it is really inefficient way of finding edge cases. I wouldn't call that basic, nor the most comprehensive option. Fuzzing at the query layer is far more computationally efficient and effective.

A truly stochastic method is more likely to hit against edge cases, rather than an agent that tends to towards idiomatic solutions and that is trained against a corpus of existing software, and burns millions of tokens/watts spinning its wheels.

^ There's plenty of business value to be found in agentic AI without reaching for it for every solution. I'd even posit agentic AI is even better when paired with focused old-fashioned squishy-brained software engineering in the loop.

[apf6]

you don't have to compare whether stochastic fuzzing or agentic testing is the best. You can just do both of them. They both have enough ROI to be worth the trouble. That's why I said "in addition to".

[cookiengineer]

Unit tests don't test for branch coverage.

That's the culprit, because LLMs tend to forget and remove a lot of branch logic in these kinds of tasks. If unit tests don't cover these specific if/elseif/else cases, then they'll just disappear.

They'll also disappear if the LLM is allowed to modify the unit tests, because they sure like to cheat their way around into greenlit test suites. The agentic environment must disallow write access to the unit test files for the agent that writes the code.

If you implement that in your tools, you'll see quickly how the models will try to rewrite the unit tests at all cost, no matter what kind of prompting you've done. Tool policies are the only boundary to successfully guarantee this.

Source: Am building my own agentic environment because of that behavior

[Keyframe]

That's the million dollar question. Do you/we/us have tests that cover everything which covers QA as well? If such a mythical beast exists, maybe from remnants of ye olde TDD past and hasn't been modified as such.. then maybe this would be possible to do as such.

[ramijames]

I've always played the game that tests are only as useful as you have resources to attribute to them. Mostly all modern development is a compromise between new features and stuff that supports those new features (tests included, but also reviews, code maintenance, docs, etc.).

If LLMs can be utilized to quickly make deep testing possible, I think that's probably a net-positive.

[jakswa]

what model did you try it with? I agree and also push back a bit: How will we know when the LLMs reach the point of handling it if no one takes the leap? I applaud more people sludging through the slop and hauling their slop buckets around.

An example is Fable being released. I felt like the most complex thing I was willing to sludge through was having it clone llama-server's web UI with my own opinions (I really like the original, kudos to them). And the initial skeleton was working so well I felt like I had sunk the tokens and committed to getting it the rest of the way: https://inkcap.click

[Keyframe]

Last time it was Opus 4.8 to no avail. Fable is too expen..precious to try that yet :)