[wrs]

Then you would expect the implementation to be structured the same as SQLite, and having glanced at the result, it looks like at least some things aren't. For example, it seems to use an operator-tree executor rather than SQLite's bytecode interpreter.

[kgeist]

The knowledge is lossy, and code generation itself is non-deterministic (temperature), so the operator-tree executor must be interference from other DB implementations, because it's uncommon to have a bytecode interpreter

[jackb4040]

The point is that the model has the source code in its weights. Therefore it's misleading to focus on being "only given the documentation", when it could produce a SQLite implementation in rust with no documentation at all. The wording is designed to create a marketing impression that it can do spec-driven development for a novel project, but this example doesn't actually support that.

It's like a diet system advertising how much weight a spokesperson lost on their system, without disclosing that they also went on Ozempic at the same time.

[mrtesthah]

To test this hypothesis you would need to look at the initial version before subsequent agents started refactoring it.

[NitpickLawyer]

Why tho? The whole point of agentic is that it takes several passes to get to a result. If step a) is a non-idiomatic transpile, but step b) c) and ... z) make it idiomatic and performant and efficient, why look at step a)? That was just the beginning. The idea with "agents" and "swarms" and dynamic this and that is that you only look at the last step.

This strictly follows the early agentic usage, where people were complaining of code full of errors and such. But having the loop do plan-write-verify is the thing that got us from that state to today. And it works. If you look at the code and it has errors (linting, etc) then your loop is not properly setup. You should only be called upon when all that can and should pass does. Your role is "direction/taste/human-common-sense-at-a-glance".

[bscphil]

Because the point of the test is to see what the agentic swarm approach is capable of. If your test assumes that a version of the thing you are trying to build is available in the training data, or can't determine if it matters whether the thing you're building is in the training data, then it's not a very useful test. Most of the time when you consider agentic approaches, you're not trying to rebuild something the agents involved are trained on.

[NitpickLawyer]

I see, that makes sense. Thank you.