| Really appreciate your perspective here! I do _not_ have a background in technical education, and am certain you've used and seen the failure modes of LLMs in this space far more than I have so far. A few thoughts based on my limited experience building and using lathe: - Part of the lathe skills are to first find source materials to base curriculum on. It's not foolproof by any means, nor is it a novel approach, but it's helped ground the content in reality more than an open ended prompt usually does (in my experience) - We're scoped to tutorials, over full blown curriculum. I found having lathe write one part of a tutorial series at a time, over the whole thing at once, usually gave me better results (and is why `/lathe-extend` is a thing) - To your point about not having the instinct to realize when it's giving me a poor progression or untrue content, my experience is that by actually writing the program the tutorial walks me through, I get definitive proof of if the results are true or not. One of the most impactful (and all too frequent) answers I got as a young programmer was "write a program and find out" and it's still good advice today. Not at all proposing this makes lathe tutorials infallible, but in the context and scope of the project it seems to take the bite out of the worst failure modes here. That said, maybe that implies lathe is most useful and least dangerous in the hands of an experience developer looking to learn a new domain, over someone looking to build foundational technical (and technical learning) skills? I'll think that over! I'm really curious what your experience would lead you to think about the above though? Are there critical failure modes for LLMs writing hands on technical education I just haven't tripped over yet? |
Context and scope limitations are also helpful, as you mention. And yes, having experience in a domain makes learning with an LLM a dramatically different experience than from-scratch, since the LLM is nudged in different directions by our responses. When a novice uses an LLM to learn, the questions they ask the LLM can drive it in directions and hallucinations that would look obvious to an experienced person.
The worst failure mode is what I mentioned: the novice asking the wrong questions or driving the LLM in the wrong direction. Inference is strongly influenced by input tokens, and that's fairly unavoidable.
I don't mean to say your project doesn't have value though! I hope people use LLMs to help them learn (by directing them to good source materials from humans) rather than just asking it to do things for them and blindly trusting the results.