[godelski]

  > I can get the exact right piece of knowledge I need to advance my understanding on demand

This is where I disagree. It would be different if these LLMs were acting as instructors and pushing you through courses designed for learning things, but this is more akin to looking at the section of a textbook that contains the exact paragraph you need. Or doing the same thing with a manual. I do not think this is the best way to learn and I actually think it is a good way to perpetuate misunderstandings. There are plenty of bad textbooks and docs, I don't want to dismiss that, but that extra information in the chapter, the previous chapters, or leading up to that paragraph are important. They are designed to be learned in order for a reason. Skipping that other information more often harms you than helps you. It gets you done with a task faster, but doesn't make you learn faster. Two different things. For review, that's different though, but the other knowledge is already there.

I think there's this belief that there are shortcuts to learning. That's a big mistake. You can't learn programming by reading, yet so many people try to do something similar with different domains. It is exactly the same thing that leads people to conspiracy theories. They have such "swiss cheese knowledge" that they don't understand how things actually are connected. How people use LLMs is typically to take the direct route to the information they want, which is only logical, but misses all that other important stuff that you wouldn't know is important to understanding those things until you have mastery of that knowledge.

If there was a shortcut, people would be writing manuals and textbooks very differently. We've been doing it for thousands of years and iterating it for just as long. It's converged to the place it has for a reason.

  > Most people have profoundly bad ideas

Yes, and why? One of the most common reasons is people are missing the surrounding context. All those little details. It's exactly what I said before about figuring it out as you go. This is part of why the "doing" matters. Why that stuff that doesn't seem important to the novice actually is, and why experts include it in their teachings.

  > you can implement and get that feedback ten times faster,

You can, but you can also dig yourself into a 10x deeper hole 10x faster. The LLM doesn't make you an expert. The LLM doesn't make context appear. I'm sorry, but all that nuance doesn't go away. The LLM only makes it less visible as it does some things for you and every time it does something you don't know how to do you're that much deeper into the water. It's okay to be in a little over your head (that's how we learn) but these tools also make it very easy to get into much deeper waters than you can handle. When that happens, you are unable to do anything and are at the mercy of the LLM. So good luck.

  > I don't think this is a coherent statement

Because complexity is complex. There are many different types. Sit with the idea longer, I promise it is coherent. But I'm not going to give you a shortcut. Maybe the LLM will, I'm fairly confident it will be able to figure it out.

[tern]

Honestly, based on where this is going, I suspect we might agree more than it appears in this short exchange.

As for the point about complexity, I will meditate on it.

My first cut is that what you're getting at is something like: you're using "complexity" in the technical sense of the word, in which case we're in the realm of so-called "wicked problems" and such. There is some kind of 'making sense' that goes beyond 'mere ideas,' or at the very least something like "throw more minds and hands at the problem," in this context, is the wrong attitude.

In such a world, a machine that amplifies human effort (while obscuring it) is not the right tool for the job—more than likely you find yourself spinning around an attractor basin.

I personally have a tentative conclusion that I specifically am able to avoid this by amplifying certain strategies I've developed over my life, but really, I don't have solid confirmation yet.

In any event, I'm enjoying the experiment, but I'll reflect on why you seem to be certain about it.

[godelski]

I agree and think we're mostly on the same page fwiw. But there are parts I disagree with. We are more aligned than misaligned. But I do want to clear some things up

  > you're using "complexity" in the technical sense of the word

Unfortunately complexity is a bit more complex than this and I think it's leading to some miscommunication. Even in technical settings complexity is overloaded. Something can be computationally complex but easy to understand. Something can be conceptually complex, but trivial to compute. I do really mean "complexity is complex".

But I am using it in both senses. I'll make a pretty straight forward argument here. As we advance on any given topic it becomes harder to advance, right? Just take something simple like playing a video game. You start from nothing and then you get a lot better simply by figuring out the controls. But as you progress towards the "professional" level it both gets computationally harder to advance (it requires more time and training) as well as conceptually harder (as little details and the number of things you need to consider increases). You have to process more information and you have to account for more information. Another way to see this is with a mathematical example: a Taylor Series. A first order approximation is relatively easy to compute but it quickly grows as you want your accuracy to increase. And the relationship isn't linear...

  > I personally have a tentative conclusion that I specifically am able to avoid this by amplifying certain strategies I've developed over my life

I can't answer this. I'm just some dude on the internet and have no idea about you. So my response shouldn't be taken in that way.

What I can do is ask an important question. Are you able to confirm you are unique or are you just tricking yourself?

Either could be true, right? I mean we put mental defenses around us all the time. There's so much crazy shit we treat as mundane. I mean just a solar flare being the size of a dozen earths and we treat it like our little minds can comprehend that and make it mundane. We like to convince ourselves that we understand things much better than we actually do. It's normal, but recognition of it is important to help us continue moving forward, right? To question what we believe. Since our understanding is never perfect, there's always more, but our brains like to feel done as that feels accomplished.

It certainly does not help that we're working with tools that are trained for human preference. I use them too. I am an AI researcher even. But I think we also need to be careful. The same objective we use to train these systems to be accurate and sound natural in their speech also maximizes their ability to deceive us. It is a really tricky thing, so I think it is also something we need to be extremely cautious about. If we don't believe we could be the fool then that makes us the biggest fool of them all, right? So how do we find out if we've been fooled or not? I don't think that's so easy and I honestly don't have an answer. There's a lot of complexity to this as you dig in and... complexity is complex.

[tern]

Yeah, I know what you mean. I do my best to check myself and also run things by people I trust, but there's an ever-present risk I'm going insane.

As a test, I've been attempting an incredibly complex project that goes far beyond my abilities as a kind of deliberate worst-case-scenario. It's more or less a programming language for a very specific purpose that compiles to a custom bytecode and runs on a custom runtime with specific performance guarantees.

I've spent part of the last month iterating on a formal model of the system and various specifications. Along the way, I teach myself how to understand and critique the part of the system I'm working on, however I also deliberately keep things just beyond my understanding by opportunistically pulling in concepts from various sources ... algebraic topology, obscure corners of PL, concepts plucked from similar systems. It's a complete monstrosity with, now, hundreds of supporting documents, research spikes, processed references, critique passes, etc.

If I'm able to complete this project and have it work as expected, I think I'll have learned a lot about what is or isn't possible. If the current design does in fact work, I'm fairly confident I'll have advanced the state of the art in the niche field I'm working in.