There will be a comeback of literate programming. Tools were not advanced enough for it to be really practical. CS was not mature enough as a field for LP to flourish.
it makes it possible for me to cope with projects which require multiple files or which exceed a certain threshold of lines/complexity.
>With LP, I can quickly look through a PDF, using hyper-linked ToC or index, and control/command click at a point and then have the editor open at that point.
I was not referring to TeX and PDF files. Donald Knuth is the creator of Literate Programming and Tex, but I personally consider this toolchain to belong to the stone age right now.
I was referring to:
1) Org mode and Emacs. LP was introduced as a concept in 1984, and org-mode was created 2007 or close to that. 20+ years later till we finally got a good formatting option for text + code. One can export to PDF or HTML, but i see no reason why not read the code directly in org-mode. It is better actually.
2) A good programming language with types. That's Rust and it was created in 2015.
3) LLM's to automatically generate descriptions of code. Type annotations are one of the most important hints for LLM's to generate accurate descriptions. Untyped or weakly typed code, like, Emacs Lisp, Python etc, does not help LLM's to be as good as possible.
4) A good LLM which is accurate enough, plus cheap enough and fast. That's Llama3-8b on Groq, in which they provide very fast access to Llama3.
I am currently researching how i can take a Rust project 100000 lines, and transform it to a Literate Programming org document. I have created a small program to cut Rust source code to chunks in an intelligent way [1]. Now i am trying to figure out, how to provide structure to the LLM, by extracting type signatures with rustdoc and feed it as context to the LLM with each query.
It will take some months before i have something good, but based on my calculations, an 100000 line program will be transformed to a giant org literate programming document, in an hour, costing close to, 50 cents.
Point being that LP will make a comeback, it's gonna be quick and it's gonna be cheap.
The project is stalled, because at some point after i developed it, the project langchain-rust appeared and gained traction. My project is not very advanced compared to langchain, i used regexes while langchain uses tree-sitter.
Langchain-rust for the moment supports only 6 languages, and not even one lisp, like Elisp, but it's tree-sitter integration means it can be extended easily to many many other languages. Also it doesn't support Groq for the moment just OpenAi, Claude and Ollama.
We are getting to Literate Programming being very practical, we are not there just yet, but it's coming soon.
I am researching also how to anticipate in changes in the code, by retrieving git logs, and invalidate some description of the code, which may have fallen behind. There are several wrinkles that need some work.
My needs/usages are far more prosaic, and I am barely a programmer, so the basic .dtx and package which makes documented .tex amenable to use with other languages and a couple of decent plain text editors is working for me.
I'd love for there to be broader support --- I considered quarto and it seemed quite promising, and I wish that LyX had a mode specifically for this, and for a while I was so desperate I was actually considering using GitBook's file inclusion feature, having all the code in discrete little files, and maintaining a series of batch files to concatenate them into working versions.
Alternately, I've been curious if there is some sort of intersection between Literate Programming and Visual Programming ---
That's a million dollar question. Can code be represented in a visual way? Bret Victor tries to give an answer to that. I don't personally think that it can, but there is sure a way, to represent some parts of a program in a visual way.
People use literate programming today in Emacs, in case you are interested. I strongly recommend to try Emacs, LLM's can write Elisp for you, to automate parts of your workflow.
https://news.ycombinator.com/item?id=40474484
it makes it possible for me to cope with projects which require multiple files or which exceed a certain threshold of lines/complexity.
>With LP, I can quickly look through a PDF, using hyper-linked ToC or index, and control/command click at a point and then have the editor open at that point.