[avidiax]

We already have effectively write-only code. It's all the assembly language and eventually machine code that we produce from other languages.

The problem with "write-only code" as it relates to LLMs is that we don't have a formal definition of the input the the LLM, nor do people typically save the requirements, both implicit and explicit, that were given the LLM to generate the code. English language will never be a formal definition, of course, but that obviously doesn't prevent the creation of a formal definition from English nor reduce the value of the informal description.

This is very similar to the problem of documentation in software development. It is difficult to enumerate all the requirements, remember all the edge cases, recall why a certain thing was done in a certain way. So computer programs are almost never well documented.

If you knew that you currently have a bunch of junior developers, and next year you will replace all of them with senior developers who could rewrite everything the junior developers did, but taking only a day, how would that affect your strategy documenting the development work and customer/technical requirements? Because that's what you have with current LLMs and coding agents. They are currently the worst that they'll ever be.

So there are two compelling strategies:

1) business as usual, i.e. not documenting things rigorously, and planning to hack in whatever new features or bugfixes you need until that becomes unsustainable and you reboot the project.

2) trying to use the LLM to produce as much and as thorough documentation and tests as possible, such that you have a basis to rewrite the project from scratch. This won't be a cheap operation at first, (you will usually do strategy #1), but eventually the LLMs and the tooling around managing them will improve such that a large rewrite or rearchitecture costs <$10k and a weekend of passive time.

[seanmcdirmid]

Is that really true though? Are people really not saving design documentation to the code repository along with the code? And is it really too much to ask in a prompt to make the LLM document aggressively? Do the LLMs you use complain about being asked to write lots of comments in their code? Is it a token cost thing? It seems ridiculous not to just start with #2 but I might be spoiled by not knowing how much my token usage costs.

[avidiax]

It requires real work to create good documentation and review it.

I've never worked in a place that requires that every commit update some documentation, but if you want to rebuild software based on the documentation, that's what it would take.

The best you could say is that the typical development process today tends to scatter documentation across commit descriptions, feature docs, design reviews, meeting notes, training materials, and source code comments.

To have a hands-off rewrite of a codebase with LLMS, you would need a level of documentation that allows a skilled human to do the rewrite without needing to talk to anyone else. I doubt that any project would have this unless it was required all along.

[seanmcdirmid]

It doesn’t though. You tell the LLM what to document up front, it can be some domain specific detail or whatever. You don’t need to carefully review it unless things go wrong. You do change propagation and change everything (tears, docs, code) in sync when some spec aspect changes, which involves a QA feedback loop, which judges the correctness of code and tests (because the LLM wrote the tests, they can be wrong). I guess you could add doc review steps as well if you knew what to look for generically, but I found it’s ok at writing docs that I haven’t needed to do this yet.

[pconstantine]

The only docs that explain what code does are a carbon copy of the code. I've saw that "no-code, write docs, write spec" movie so many times, and it always ends as a flop in the box office.

[pconstantine]

It will never work. Code is the documentation. The difference between Code -> IR -> binary is that the transition is deterministic. Doc -> Code is non-deterministic asf. And will never be. The only reason it works and we have senior, middle, junior engineers is the fact that assumptions on that path that senior engineers make are better than the ones junior engineers make. LLMs cannot make any assumptions though, as they have zero real world knowledge and thus zero common sense.