[simonw]

Bun had an enormous existing test suite written in TypeScript. Getting those tests to pass against the Rust version was the key thing that enabled the project.

You can go and check if the AI edited the tests yourself: look at the git history of those files in the public Bun repository.

[skylurk]

But Bun also implements the interpreter to those tests— doesn't that mean it has a lot of leeway to over-fit to them? It's not as clean as "don't change these files" in this case.

In general I would agree though, and would take it one step further: the agent writing the source code should not even be able to see the tests, nor the agent writing the tests see the source code.