[wiseowise]

> Having a CI job that identifies places where the docs have drifted from the implementation seems pretty valuable.

https://docs.python.org/3/library/doctest.html

> To check that a module’s docstrings are up-to-date by verifying that all interactive examples still work as documented. To perform regression testing by verifying that interactive examples from a test file or a test object work as expected. To write tutorial documentation for a package, liberally illustrated with input-output examples. Depending on whether the examples or the expository text are emphasized, this has the flavor of “literate testing” or “executable documentation”.

Seems pretty related to me.

[tonyarkles]

I appreciate doctest very much, but those aren’t the kind of documentation I’m worried about drifting. I’m thinking more on the “how does this communication protocol between this server and this client work?”, which is generally terrible to try to summarize in doctests. If you want to take the idea to the extreme, imagine a CI test that answers “does this server implementation conform to this RFC?”

[1st1]

Not really.

> Having a CI job that identifies places where the docs have drifted from the implementation seems pretty valuable.

Testing with lat isn't about ensuring consistency of code with public API documentation. It is about:

* ensuring you can quickly analyze what tests were added / changed by looking at the English description

* ensuring you spot when an agent randomly drops or alters an important functional/regression tests

The problem with coding agents is that they produce enormous diffs, and while reading tests code is very important in practice your focus and attention drifts and you can't do thorough analysis.

This isn't a new problem though, the same thing applies to classic code reviews -- rarely coding is a bottle neck, it's getting all reviews from humans to vet the change.

Lat shifts the focus from reading test code to understanding the semantics of the test. And because instead of reviewing 2000 lines of code you can focus on reviewing only 100 lines change in lat.md you'll be able to control your tests and implementation more tightly.

For projects where code quality isn't paramount I now just glance over the code to spot anti-pattern and models failing to DRY and resorting to duplicating large swaths of code.