[tenzo]

we've faced this same problem but thinking about it a little deeper reminded me that this issue is no different to pre-llm world. someone creates a spec, someone or a team build/implement the spec and after actioning any feedback from testing or from customers, the product evolves and is no longer aligned with the initial spec.

i think one way to solve this is having an someone or an agent update the initial spec with any new changes as and when they are made.

[al_borland]

I’ve seen at least one person where I work doing this. They have an agent doing massive updates on Confluence constantly. It seems like a mess. It checks the box for having documentation, if that’s all anyone cares about. But there is no way people are actually reading it, looking for changes is impossible, since the LLM re-writes the entire page every time, and even not knowing their project well, it was full of hallucinations and nonsense.

If the LLM is going to write the doc anyway, why not just do it on demand from the code if/when it’s needed, instead of burning tokens on documents that no one will ever read?

I also think a lot of the value in those historical docs are the reasons why certain decisions were made, as it can be important context if there is every a discussion on doing things differently. The LLM likely won’t have that information, or may just make up a bunch of reasons the business doesn’t actually care about.