[emporas]

I was not referring to TeX and PDF files. Donald Knuth is the creator of Literate Programming and Tex, but I personally consider this toolchain to belong to the stone age right now.

I was referring to:

1) Org mode and Emacs. LP was introduced as a concept in 1984, and org-mode was created 2007 or close to that. 20+ years later till we finally got a good formatting option for text + code. One can export to PDF or HTML, but i see no reason why not read the code directly in org-mode. It is better actually.

2) A good programming language with types. That's Rust and it was created in 2015.

3) LLM's to automatically generate descriptions of code. Type annotations are one of the most important hints for LLM's to generate accurate descriptions. Untyped or weakly typed code, like, Emacs Lisp, Python etc, does not help LLM's to be as good as possible.

4) A good LLM which is accurate enough, plus cheap enough and fast. That's Llama3-8b on Groq, in which they provide very fast access to Llama3.

I am currently researching how i can take a Rust project 100000 lines, and transform it to a Literate Programming org document. I have created a small program to cut Rust source code to chunks in an intelligent way [1]. Now i am trying to figure out, how to provide structure to the LLM, by extracting type signatures with rustdoc and feed it as context to the LLM with each query.

It will take some months before i have something good, but based on my calculations, an 100000 line program will be transformed to a giant org literate programming document, in an hour, costing close to, 50 cents.

Point being that LP will make a comeback, it's gonna be quick and it's gonna be cheap.

[1]https://github.com/pramatias/documentdf

[WillAdams]

It is somewhat ironic that that GitHub link has no visible documentation when one first opens it.

[emporas]

The project is stalled, because at some point after i developed it, the project langchain-rust appeared and gained traction. My project is not very advanced compared to langchain, i used regexes while langchain uses tree-sitter.

Langchain-rust for the moment supports only 6 languages, and not even one lisp, like Elisp, but it's tree-sitter integration means it can be extended easily to many many other languages. Also it doesn't support Groq for the moment just OpenAi, Claude and Ollama.

We are getting to Literate Programming being very practical, we are not there just yet, but it's coming soon.

I am researching also how to anticipate in changes in the code, by retrieving git logs, and invalidate some description of the code, which may have fallen behind. There are several wrinkles that need some work.

[WillAdams]

Interesting! Thanks for the explanation.

My needs/usages are far more prosaic, and I am barely a programmer, so the basic .dtx and package which makes documented .tex amenable to use with other languages and a couple of decent plain text editors is working for me.

I'd love for there to be broader support --- I considered quarto and it seemed quite promising, and I wish that LyX had a mode specifically for this, and for a while I was so desperate I was actually considering using GitBook's file inclusion feature, having all the code in discrete little files, and maintaining a series of batch files to concatenate them into working versions.

Alternately, I've been curious if there is some sort of intersection between Literate Programming and Visual Programming ---

>What does an algorithm look like?

[emporas]

>What does an algorithm look like?

That's a million dollar question. Can code be represented in a visual way? Bret Victor tries to give an answer to that. I don't personally think that it can, but there is sure a way, to represent some parts of a program in a visual way.

People use literate programming today in Emacs, in case you are interested. I strongly recommend to try Emacs, LLM's can write Elisp for you, to automate parts of your workflow.

[WillAdams]

Will look up Bret Victor --- thanks!

Unfortunately, while I like emacs-style keyboard bindings in Mac OS X, I prefer to use a stylus and often write things out by hand and use handwriting recognition to convert to text (hence my interest in visual programming languages) --- very much envied a co-worker (who was one of two folks I've met aside from TeX conferences who had read TAoCP) who just started it up at the beginning of the day and got all his work done in it during the course of the day.