[simonw]

I don't think building it the long way is necessarily a more effective way to learn.

You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works.

My hunch is that the 30 minutes of focused learning spent with a custom-built version that solves your exact problem is as effective (or even more effective) than four hours spent mostly struggling to get something up and running and going down various rabbit holes of unrelated problem-solving.

Especially if realistically you were never going to carve out those four hours anyway.

[aabhay]

This feels like the exactly wrong way to think about it IMO. For me “knowledge” is not the explicit recitation of the correct solution, it’s all the implicit working knowledge I gain from trying different things, having initial assumptions fail, seeing what was off, dealing with deployment headaches, etc. As I work, I carefully pay attention to the outputs of all tools and try to mentally document what paths I didn’t take. That makes dealing with bugs and issues later on a lot easier, but it also expands my awareness of the domain, and checks my hubris on thinking I know something, and makes it possible to reason about the system when doing things later on.

Of course, this kind of interactive deep engagement with a topic is fast becoming obsolete. But the essence to me of “knowing” is about doing and experiencing things, updating my bayesian priors dialectically (to put it fancily)

[simonw]

I agree that the only reliable way to learn is to put knowledge into practice.

I don't think that's incompatible with getting help from LLMs. I find that LLMs let me try so much more stuff, and at such a faster rate, that my learning pace has accelerated in a material way.

[gflarity]

Consider, ever so briefly, that people don't all learn the same. You do you.

[simonw]

That's fair.

Something I'm really interested right now is the balance in terms of the struggle required to learn something.

I firmly believe that there are things where the only way to learn how to do them is to go through the struggle. Writing essays for example - I don't think you can shortcut learning to write well by having an LLM do that for you, even though actually learning to write is a painful and tiresome progress.

But programming... I've seen so many people who quit learning to program because the struggle was too much. Those first six months of struggling with missing semicolons are absolutely miserable!

I've spoken to a ton of people over the past year who always wanted to learn to program but never managed to carve out that miserable six months... and now they're building software, because LLMs have shaved down that learning curve.

[theLiminator]

I think it really depends on how it's used. It's a massive accelerant if it's just helping you stitch stuff together. Or when it helps you get unblocked by quickly finding you the missing api you need.

But when it replaces you struggling through figuring out the mental model of what you're doing, then I think you end up learning at a much more shallow level than you would by doing thing manually.

[AmbroseBierce]

And that's a fast lane for security issues a plenty, when you cannot spot them because you don't even understand what each part is supposed to do.

[habinero]

That's not learning, that's building. It's like trying to learn how to draw via paint by numbers. Do you end up with something you could hang on the wall? Sure. Could you have fun doing it? Sure. Is there anything wrong with just doing that as a hobby? Of course not.

Is it a substitute for actually learning how to look at objects and break them down into shapes and color and value? No. You gotta put in the work if you want the result. Brains just work like that.

"Struggling with semicolons" isn't any different than drawing a hundred derpy looking faces that look terrible.

Ira Glass has a quote about this:

> “Nobody tells this to people who are beginners, I wish someone told me. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase, they quit. Most people I know who do interesting, creative work went through years of this. We know our work doesn’t have this special thing that we want it to have. We all go through this. And if you are just starting out or you are still in this phase, you gotta know its normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story. It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions. And I took longer to figure out how to do this than anyone I’ve ever met. It’s gonna take awhile. It’s normal to take awhile. You’ve just gotta fight your way through.”

[Herring]

I like the sentiment, I really do, but nobody (outside a phd program) pays you to learn. That's just not how society is set up. If FAANG companies could get away with hiring high school kids at min wage to prompt all day they would. We'll figure that out real quick as that exponential rises. If you don't like it, build a better society. While you still can.

[simonw]

I love that Ira Glass quote. I've thought about it a lot!

I still think paint by numbers is a valid early step along the path to learning to draw.

[mmasu]

I remember a very nice quote from an Amazon exec - “there is no compression algorithm for experience”. The LLM might as well do wrong things, and you still won’t know what you don’t know. But then, iterating with LLMs is a different kind of experience; and in the future people will likely do that more than just grinding through the failure of just missing semicolons Simon is describing below. It’s a different paradigm really

[visarga]

Of course there is - if you write good tests, they compress your validation work, and stand in for your experience. Write tests with AI, but validate their quality and coverage yourself.

I think the whole discussion about coding agent reliability is missing the elephant in the room - it is not vibe coding, but vibe testing. That is when you run the code a few times and say LGTM - the best recipe to shoot yourself in the foot no matter if code was hand written or made with AI. Just put the screw on the agent, let it handle a heavy test harness.

[mmasu]

this is a very good point, however the risk of writing bad or non extensive tests is still there if you don’t know what good looks like! The grind will still need to be there, but it will be a different way of gaining experience

[DANmode]

Starting to get it!

New skills, not no skills.

There will still be a wide spectrum of people that actually understand the stack - and don’t - and no matter how much easier or harder the tools get, those people aren’t going anywhere.

[barrkel]

Compression algorithms for experience are of great interest to ML practitioners and they have some practices that seem to work well. Curriculum learning, feedback from verifiable rewards. Solve problems that escalate in difficulty, are near the boundary of your capability, and ideally have a strong positive or negative feedback on actions sooner rather than later.

[johnfn]

But how much of that time is truly spent on learning relevant knowledge, and how much of it is just (now) useless errata? Take vector search for an example. Pre-GPT, I would spend like an hour chasing down a typo, like specifying 1023 instead of 1024 or something. This sort of problem is now trivially solved in minutes by a LLM that fully understands the API surface area. So what exactly do I lose by not spending that hour chasing it down? It has nothing to do with learning vector search better, and an LLM can do it better and faster than I can.

[extr]

I think people fool themselves with this kind of thing a lot. You debug some issue with your GH actions yaml file for 45 minutes and think you "learned something", but when are you going to run into that specific gotcha again? In reality the only lasting lesson is "sometimes these kinds of yaml files can be finnicky". Which you probably already knew at the outset. There's no personal development in continually bashing your head into the lesson of "sometimes computer systems were set up in ways that are kind of tricky if you haven't seen that exact system before". Who cares. At a certain point there is nothing more to the "lesson". It's just time consuming trial and error kind of gruntwork.

[Applejinx]

Github Actions, web development, stuff like that, are terrible examples of where not to use AI.

You can't really go to giant piles of technical debt and look to those for places to be human. It's soul-destroying. My concern would be that vibe coding will make those places of soul-less technical debt even deeper and deadlier. There will be nobody there, for generations of cruft. Where once the technical debt was made by committee, now it'll be the ghosts of committees, stirred up by random temperature, only to surface bits of rot that just sink down into the morass again, unfixed.

When 'finicky' is actually an interesting problem, or a challenge, that's one thing. When 'finicky' is just 'twelve committees re-hacked this and then it's been maintained by LLMs for years', there is nothing gained by trying to be human at it.

[iwontberude]

I have a friend that took over a project that another dev started that had literally hundreds of markdown documents in repo with things as insane as software for managing souls in a quantum immortality scheme.

[iwontberude]

I don’t think it foolishness. Through random sampling (troubleshooting problems) you can construct a statistically significant model for understanding the whole of the problem space. Maybe it doesn’t scale linearly with the amount of samples but it’s additive for sure.

[jstummbillig]

I think is exactly right in principle and practically. The question is what domain knowledge you should improve on to maximize outcome: Will understanding the machine code be the thing that most likely translates to better outcomes? Will building the vector search the hard way be? Or will it be focusing on the thing that you do with the vector search?

At some point things will get hard, as long as the world is. You don't need to concern yourself with any technical layer for that to be true. The less we have to concern ourselves with technicalities, the further that points shifts towards the thing we actually care about.

[PessimalDecimal]

Forgetting LLMs and coding agents for a second, what OP describes is like watching a Youtube video on how to make a small repair around the house. You can watch that and "know" what needs to be done afterwards. But it is a very different thing to do it yourself.

Ultimately it comes to whether gaining the know how through experience is worth it or not.

[viking123]

It's like reading a math book.

[grim_io]

Trial and error is not how apprenticeship works, for example.

As an apprentice, you get correct and precise enough instructions and you learn from the masters perfection point downwards.

Maybe we have reached a point where we can be the machine's apprentices in some ways.

[gtowey]

You could say that knowledge is understanding all the paths that won't solve your problem.

[bulbar]

Take a look at Bloom's taxonomy. It's exactly about what you are talking about.

[throwaway613745]

Just speaking from personal experience but the struggle is what creates the learning.

I learned refactoring patterns from Fowler's book. But when I tried to actually use them I still struggled. I didn't fully understand how the patterns worked until I actually tried (and failed) to use them a few times.

You don't really internalize things until you understand what doesn't work just as much as what does. You don't learn nearly as much from success as you do from failure. I would say the ratio of truly internalized knowledge is much higher for failure.

The notion that you can get a bot to just vomit out a vector database and then you can just "read the code" and you'll understand how a vector database works is just ludicrous.

[simonw]

This conversation isn't about building a vector database from scratch, it's about learning to integrate with an existing vector database.

[throwaway613745]

The topic is basically irrelevant. I could just edit my post to change the two instances of "vector database" to "vector database integration" and nothing else would change about my point.

I could change the post to be about learning word-working by watching a robot build a shelf and nothing would change.

[simonw]

I genuinely do think you can learn 90% of what that is to learn about integrating with a vector database from having an LLM do the work for you and then carefully reviewing what it did.

Turns out there's science that backs me up here: https://en.wikipedia.org/wiki/Worked-example_effect - showing people "worked examples" can be more effective than making them solve the problem themselves.

That Wikipedia article is a little weak, this MIT page is better: https://tll.mit.edu/teaching-resources/how-people-learn/work...

[throwaway613745]

> Worked examples are step-by-step illustrations of the process required to complete a task or solve a problem.

That’s not what having a bot generate your integration is and reading it post-facto is. The bot isn’t guiding you through the process so you can go do it yourself. At best you would use this as a reference to go do another integration yourself - but at this point why even bother when you can just get the bot to do it again?

The only thing people learn using AI is how to do things with AI.

[simonw]

> The bot isn’t guiding you through the process so you can go do it yourself.

It is if you ask it to. Learning well with LLMs requires a lot of self-discipline - you have to be actively aware of the threat that you won't actually learn effectively and take steps to counter that.

I keep meticulous notes of everything these things do for me, which adds up to a valuable set of notes over time. I gave up on remembering things without notes a long time ago!

[jfreds]

Having gone though exactly this exercise recently comparing a homegrown vector db against Qdrant, I’m wholeheartedly in agreement that getting a working solution FAST, and then spending a decent amount of time interrogating it (with help of LLM), is my favorite learning pattern

[girvo]

> Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works

I can only speak for myself, but the only way I've been able to learn things rapidly in this industry is by writing things myself: even rote re-typing of books or SO answers was enough to trigger this for me.

Just querying models and reading output doesn't seem to work for me, but that's maybe down to my particular learning style.

[simonw]

That's why I said "tweaking it yourself" - that's the point where you go beyond "just querying models and reading output".

[girvo]

That hasn't been enough for me, so far in my experience. I think I'm too crusty and set in my ways after 25 years of learning programming.

I have found them useful for general explanations and it's decent at finding me sources or directly answering questions about codebases/architecture (like where the @appendNode declarative mutation directive is fired in sequence for the Relay updater system), but it's code output I've not found a good teach tool for myself.

[gambiting]

>>My hunch is that the 30 minutes of focused learning spent with a custom-built version that solves your exact problem is as effective

My hunch is the exact opposite of this. You will learn close to nothing by reading this for 30 minutes.

[barrkel]

I don't know. I built a vector similarity system for my hobby project the "hard" way, which was mostly getting Python set up with all the dependencies (seriously, Python dependency resolution is a non-trivial problem), picking a model with the right tradeoffs, installing pgvector, picking an index that optimized my distance metric, calculating and storing vectors for all my data, and integrating routes and UI which dispatched ANN search (order by / limit) to my indexed column. I also did some clustering, and learned something of how awkward it is in practice to pick a representative vector for a cluster - and in fact you may want several.

I now know what the model does (at a black box level) and how all the parts fit together. And I have plans to build classifiers on top of the vectors I built for further processing.

The experience of fighting Python dependencies gives me more appreciation for uv over venv and will leave me less stuck whenever the LLM fails to help resolve the situation.

[politelemon]

That's assuming everyone learns the same way, which isn't true. Watching a streamer beat a dark souls boss won't automatically make you competent at the game. Reading through gobs of code generated for you without knowing why various things were needed won't help either. A middle approach could be to get the LLM to guide you through the steps.

[mbbutler]

It's not just assuming that everyone learns the same way. It's assuming that everyone learns the way that all of the research literature on learning claims does not work.

Learning requires active recall/synthesis. Looking at solved examples instead of working them yourself does not suffice in math, physics, chemistry, or CS, but somehow it is supposed to work in this situation?

[ktzar]

It's the same hunch we all have when we think we're going to learn something by watching tutorials. We learn by struggling.

[beasthacker]

I buy the productivity argument, but I’m not convinced “30 minutes reading/tweaking agent output” is equivalent for learning to building it yourself.

If your goal is the feature, then yes: letting the agent do the heavy lifting and reviewing the diff afterward is a huge win.

But if your goal is understanding / skill-building, the hard part usually isn’t seeing a working solution. It’s doing the messy work of (a) making design choices, (b) getting stuck, (c) debugging, and (d) forming the mental model that lets you reproduce it later. Reviewing a correct implementation can create a feeling of “I get it,” but that feeling often doesn’t survive a blank file.

I’ve noticed this in my own hobby coding: LLMs are great for familiarity and unblocking progress, but the learning “sticks” much more when I’ve had to struggle through the failure modes myself. I’m watching the same dynamic play out with my son using ChatGPT to study for physics/calculus . . . it feels deep for him in the moment with the LLM, but exam-style transfer exposes the gaps.

[simonw]

If I had four hours to dedicate to this particular learning project I would still use LLMs to help me along the way, with the expectation that I'd learn more from those four hours than I would if I'd spent the same amount of time deliberately not using LLMs to help me.

We've been given a tool that lets us ask questions in human language and get back answers that are correct 90% of the time! And that remaining 10% means we have to engage critically with those answers, which is a useful learning trick in its own right.

[beasthacker]

Hypothetical for you:

Learn more if you tried to figure it out yourself for 3 hours then used the LLM for the last hour to unblock/check your work? Or learn more by utilizing LLM for help the whole four hours?

My own experience is what I learn from an LLM sticks better if I take the former approach.

[simonw]

Depends on the task and my goals. If it was something new to me that I wanted to learn really deeply - and I had the four hours to spend - I might try the LLM-free route for the first three hours like you suggest.

If I found myself needing to do anything unrelated to the learning task, like knock out a quick Bash script, I'd still call on the LLM to get me out of that and help me stay focused on the new skill though.

[risyachka]

Reading without actually doing does not really result in learning, only very marginal one.

Try reading tutorials on a new programming language for 30 minutes and then open new text file and write basic loop with print.

It won’t even compile- which shows you haven’t really learned anything. Just read an interesting story. Sure you pita few bits here and there but you still don’t know how to do even the moat basic thing.

[simonw]

Working with an LLM feels very different to me from reading a static tutorial.

It's more like having the tutorial author there with you and actively engaging with them to collaborate on building the exact tutorial for the exact project you're looking to build.

I'll take a bidirectional conversation with a subject matter expert (and when you're just staring to learn Rust LLMs can absolutely take the role of "expert", in comparison to you at least) over struggling on my own against static documentation and the Rust compiler.

And I can take over the wheel at any moment! It's entirely on me to decide how much I get to do vs how much the LLM does for me.

That's why learning in this way is a skill in its own right, and one that I'd like to see studied and formalized and taught to other people.

[Applejinx]

This really makes for a good natural experiment: carry on :)

I have a hard time imagining how much you'd have to literally bribe me to get me to try doing it the way you describe. I'm too interested in implementation details of things and looking for innovations—in fact I make my living doing that, like some cyberpunk gremlin just delighting in messing with stuff in unexpected ways. I don't understand why you're not, but maybe it's not for me to understand.

Carry on. We'll check back and see how it worked for ya :)

[simonw]

I delight in messing with stuff in unexpected ways, and see that as a great way to learn. I just don't want to have to type all of that mischief out by hand.

I ported a complex new Python program I wrote to Go last night on my phone out of nothing more than wild curiosity. You can bet I learned a bunch about Go in the process, from bed, in about 20 minutes.

[weitendorf]

Generally I agree with your takes and find them very reasonable but in this case I think your deep experience might be coloring your views a bit.

LLMs can hurt less experienced engineers by keeping them from building an intuition for why things work a certain way, or why an alternative won't work (or conversely, why an unconventional approach might not only be possible, but very useful and valuable!).

I think problem solving is optimization in the face of constraints. Generally using LLMs IME, the more you're able to articulate and understand your constraints, and prescriptively guide the LLM towards something it's capable of doing, the more effective they are and the more maintainable their output is for you. So it really helps to know when to break the rules or to create/do something unconventional.

Another way to put it is that LLMs have commodified conventional software so learning when to break or challenge convention is going to be where most of the valuable work is going forward. And I think it's hard to actually do that unless you get into the weeds and battle/try things because you don't understand why they won't work. Sometimes they do

[simonw]

I think it's very easy to harm your learning by leaning into LLMs.

What I don't believe is that it HAS to be like this. Maybe it's my natural optimism showing through here, but I'm confident it's possible to accelerate rather than slow down your learning progress with LLMs, if you're thoughtful about how you apply them.

An open question for me is how feasible it is to teach people how to teach themselves effectively using this new technology.

I have a core belief that everything is learnable, if people are motivated to learn. I have no idea how to help instill that motivation in people who don't yet have it though!

[Arainach]

> An open question for me is how feasible it is to teach people how to teach themselves effectively using this new technology.

It's not really an open question. We've had a huge amount of content on the internet including documentation, tutorials, example code, and actual online courses available for years and in the end most people don't learn effectively when presented with that information and left to themselves. LLMs are no different.

[DANmode]

What would you have us do, though?

Stifle the tools, somehow?

You’ve had nontechnical devs since npm, or before!

No: people that care to understand the whole stack, and be able to provide that value, will still exist and shine.

[throw1235435]

> No: people that care to understand the whole stack, and be able to provide that value, will still exist and shine.

I hope so. But I don't believe so. I think us SWE's will find a way to disrupt that too as we all rush for the exits before this industry sinks.

The biggest barrier previously to anything (not just SWE) was the fact that like everything worth it in life; it takes work to see results. Generally people are time/resource poor and have to spend their own time or outsource the effort to get something - which limits the things they can do.

AI takes that away for SWE relative to other fields. People can get instant gratification now and "do it themselves" and given cost/benefit will prefer other fields now and want to spend their time elsewhere. At scale there will still be jobs for things people don't want to manage themselves but they will be more routine and busy work - not high salary skills based.

[enraged_camel]

Agree completely. The other aspect for me is that LLMs make me unafraid to take on initiatives in areas I know nothing about and/or am uninterested in pursuing due to discrepancy in effort vs reward. As a result I end up doing more and learning more.

[csomar]

The struggle is how you learn. I think that’s pretty much established scientifically by now?

[simonw]

If it is I'd very much like to learn more about the science.

I find it hard to believe that wasting hours hunting for a missing semicolon (at the very real risk of quitting entirely) is essential for learning. Does that mean every student who asks a TA or fellow-student to help them find that semicolon is hurting themselves when they do that?

If not, what's different about asking an LLM?

[simonw]

I had Claude go dig up some science for me: https://claude.ai/share/2dc95280-ff92-4b13-816f-24f5993d8fc7

The most relevant concepts appear to be:

- Desirable Difficulties - https://en.wikipedia.org/wiki/Desirable_difficulty - "A desirable difficulty is a learning task that requires a considerable but desirable amount of effort, thereby improving long-term performance. [...] The task must be able to be accomplished. Too difficult a task may dissuade the learner and prevent full processing."

- Worked-example effect - https://en.wikipedia.org/wiki/Worked-example_effect - "Specifically, it refers to improved learning observed when worked examples are used as part of instruction, compared to other instructional techniques such as problem-solving. [...] However, it is important to note that studying [worked examples] loses its effectiveness with increasing expertise"

- Expertise reversal effect - https://en.wikipedia.org/wiki/Expertise_reversal_effect - "The expertise reversal effect refers to the reversal of the effectiveness of instructional techniques on learners with differing levels of prior knowledge."

- "Generation effect" - https://en.wikipedia.org/wiki/Generation_effect - "The generation effect is a phenomenon whereby information is better remembered if it is generated from one's own mind rather than simply read."

[dns_snek]

Well that's a gross oversimplification of the process. Hunting for a missing semicolon is a basic mechanical task that doesn't require much thought.

Engaging with an intellectual problem, trying to solve it one way, failing, reasoning through the process and the requirements, trying to discover a better way of solving something, going down some wrong paths, backtracking, merging diverging ideas and ultimately finding a solution is going to yield an infinitely deeper understanding of the problem, what works, what doesn't, and improve your general intuition and problem-solving skills.

Deep engagement builds deep understanding, shallow engagement builds shallow understanding. There's no substitute for doing the hard work yourself - I've tutored classmates in school and I find this rather obvious. A tutor (human or LLM) can try to find a way to explain something in a way that you understand but if you don't do most of the hard work yourself it's never going to stick. I noticed that when I would spoon-feed answers to people it would always just lead them into a false sense of confidence.

[simonw]

My argument here is that you can still do hard work that helps you learn while leaning on an LLM to help along the way.

There's a reason kids do better when assigned a 1-1 tutor. LLMs, used effectively, can have a similar effect. Probably a weaker effect although maybe it can be stronger since there's no shame involved in asking an LLM a question.

[csomar]

I am far from being an expert on this topic. I took this course (many years ago?) https://www.coursera.org/learn/learning-how-to-learn and held this idea since.

> ind it hard to believe that wasting hours hunting for a missing semicolon

That's not what I meant by "struggle" and you do not use an LLM for that anyway.

[simonw]

What did you mean by struggle?

I use LLMs to help me spot mistakes like that all the time, and I encourage people learning to code to do the same.

[sorokod]

You can spend 30 min, watching someone learning how to ski, you will learn something. You will not be able to ski by yourself though.

[scotty79]

If you are not failing you are barely learning anything.