| What do you mean with "how all this works"? Do you want to know how program code works, or how LLMs turn natural language into code? For the former, I think some hand coding will get the best results. A large part of "thinking like a programmer" is thinking about the sort of invisible structures that code will produce in a computer's memory when it runs - which is not the same as what's literally written in the code file. IMO, the best way to get a feel for that is to write code yourself, run it, examine the results, etc. But writing code can be a rather dry activity if you don't have motivation for what to write. The usual vibe coded projects are probably too complex to hand code without experience. I think two good starting points could be Code Katas [1] or the Pico8 game engine [2] (and of course any good "intro to Python" or "intro to JavaScript" tutorial on the web) -- For the latter, you could maybe start with the history of generative text models (like markov chains) and neural networks. Both technologies are several decades old and sort of form the very basic building blocks that made development of LLMs possible. In particular, neural networks made use of the gradient descent and backpropagation algorithms that are still the basis of most LLM training processes - even if they might now run for months and take exabytes of training data. Then there is of course the "Attention is all you need" paper that introduced the Transformer architecture - the kind of latest-generation neural network that underpins most LLMs - and basically started the whole LLM revolution. (It's also the "T" in GPT) (For historical interest, there is also the whole theory of formal languages/grammars and automata - which was the previous attempt to understand human language with computers. That theory gave us a lot of things - such as the Turing Machine and the entire theory of how to build and read a programming language. What it didn't give us though was a way to process human language...) -- As for the specific question of how an LLM can take this request in human language and generate that code from it that happens to fullfill all the requirements - that's an active area of research and I honestly doubt that right now anyone understands that in all entirety. Through I believe Anthropic and LessWrong had some interesting blog posts about this topics where they tried to understand the functions of certain specific neurons in an LLM. See also "Golden Gate Claude". [1] http://codekata.com/ [2] https://www.lexaloffle.com/pico-8.php |