[theshrike79]

Simon Willison had this idea of "Documentation unit tests" in 2018: https://simonwillison.net/2018/Jul/28/documentation-unit-tes...

It's not a massively complex AI monstrosity (it's from 2018 after all) or a perfect solution, but it's a good jumping off point.

With a slight sprinkling of LLM this could be improved quite a bit. Not by having the agent write the documentation necessarily, but for checking the parity and flagging it for users.

For example a CI job that checks that relevant documentation has been created / updated when new functionality is added or old one is changed.

[dec0dedab0de]

interesting that they don’t mention doctest which has been a python built-in for quite a while.

It allows you to write simple unit tests directly in your doc strings, by essentially copying the repl output so it doubles as an example.

combined with something like sphinx that is almost exactly what you’re looking for.

doctest kind of sucks for anything where you need to set up state, but if you’re writing functional code it is often a quick and easy way to document and test your code/documentation at the same time.

https://docs.python.org/3/library/doctest.html

[theshrike79]

Doctest is writing unit tests in doctstrings.

That system is an unit test that checks that functions are documented in the documentation. Nothing to do with docstrings.

[dec0dedab0de]

right but docstrings are documentation, so if your doctest is working, then at least that part of the documentation is correct.

Even without doctest, generating your documentation from docstrings is much easier to keep updated than writing your documentation somewhere else, because it is right there as you are making changes.