[TonyAlicea10]

There is of course a degree of true usefulness to this. However I’ve been a technical educator for years and I’ve tried to do lots with LLMs.

Even now, LLMs are terrible educators. They do not make coherent progressive curriculums. They hallucinate details which the student will not have the knowledge to challenge.

If you use an LLM to make a tutorial you will get some benefit for sure, especially if you use it for Socratic sessions based on a corpus of data you provide (like a blog post or documentation).

Don’t expect it to teach you reliably though. It feels good to ask the LLM whatever you want, but if you’re learning a topic you don’t have the instinct to realize when it’s giving you a poorly chosen progression of information or teaching you something flat out untrue.

[devenjarvis]

Really appreciate your perspective here! I do _not_ have a background in technical education, and am certain you've used and seen the failure modes of LLMs in this space far more than I have so far.

A few thoughts based on my limited experience building and using lathe:

- Part of the lathe skills are to first find source materials to base curriculum on. It's not foolproof by any means, nor is it a novel approach, but it's helped ground the content in reality more than an open ended prompt usually does (in my experience)

- We're scoped to tutorials, over full blown curriculum. I found having lathe write one part of a tutorial series at a time, over the whole thing at once, usually gave me better results (and is why `/lathe-extend` is a thing)

- To your point about not having the instinct to realize when it's giving me a poor progression or untrue content, my experience is that by actually writing the program the tutorial walks me through, I get definitive proof of if the results are true or not. One of the most impactful (and all too frequent) answers I got as a young programmer was "write a program and find out" and it's still good advice today. Not at all proposing this makes lathe tutorials infallible, but in the context and scope of the project it seems to take the bite out of the worst failure modes here. That said, maybe that implies lathe is most useful and least dangerous in the hands of an experience developer looking to learn a new domain, over someone looking to build foundational technical (and technical learning) skills? I'll think that over!

I'm really curious what your experience would lead you to think about the above though? Are there critical failure modes for LLMs writing hands on technical education I just haven't tripped over yet?

[TonyAlicea10]

Source materials is great. Having the LLM write one part of a tutorial prevents you from asking it for a progressive curriculum which helps. If I give an LLM a sequence I want to learn, or an outline, it does much better.

Context and scope limitations are also helpful, as you mention. And yes, having experience in a domain makes learning with an LLM a dramatically different experience than from-scratch, since the LLM is nudged in different directions by our responses. When a novice uses an LLM to learn, the questions they ask the LLM can drive it in directions and hallucinations that would look obvious to an experienced person.

The worst failure mode is what I mentioned: the novice asking the wrong questions or driving the LLM in the wrong direction. Inference is strongly influenced by input tokens, and that's fairly unavoidable.

I don't mean to say your project doesn't have value though! I hope people use LLMs to help them learn (by directing them to good source materials from humans) rather than just asking it to do things for them and blindly trusting the results.

[protocolture]

Part of why old tech articles are so instructive, is that they tend to have 1 command or feature thats out of date. You need to then go off and figure out how to resolve the missing piece of the tutorial yourself. I see this as a likely common feature of LLMs

[tmountain]

I have been working on a language learning app for myself, and I am using a textbook that I like as the basis for an Anki inspired “learning tree”. This is working pretty well because I can build progressions from the original table of contents.

[TonyAlicea10]

Yes, generally if you get a decent progression from an LLM, I find it's copying directly from a source, like a book or course. Giving it a progression helps a lot.