[krackers]

> it’s only when you fully understand the problem you can generalize

Peter Naur explained this decades ago

>Peter Naur argues that programming is fundamentally a human activity of building a mental "theory" - a deep conceptual insight into how a system's parts match the real-world problem it solves. He rejects the prevailing view that programming is merely the mechanical production of source code, specifications, and documentation. Instead, Naur posits that the true product of programming is the shared mental model held in the minds of the developers who built it.

As a human you're the one with the problem you want code to solve, so it's worth having an understanding the problem. Otherwise you risk an X/Y situation, where the LLM ends up solving a problem that may not actually satisfy what you need.

I think what LLMs allow you to do is better abstract away everything that's _not_ essential to the problem you care about, in the same way that libraries or any higher-level programming language does. Ultimately there's still a "core" of the problem that needs to be expressed formally though. When you see people "vibe coding" by prompting the LLM to add constraints at a time until they reach their desired end-goal, this is ultimately "programming" in a sloppy, non-formal way. Better to get the LLM to write everything else _surrounding_ the problem, so you can write and understand the core yourself.

[balder1991]

Basically, I’d say the LLM doesn’t have a “theory” of anything that wasn’t in its training. It’ll always treat your project as if it was reading a book for the first time.

As an example, if you chat with an LLM it can sort of explain concepts it learned in its training, because it created some internal representation of them. However it’ll never be able to explain things about your project unless you tell it (then it can simply repeat it, which is the same RAG does, the LLM is just spitting it out without any deep understanding of it).

You can try to cram as much context to it as possible and overload it with extra information, or you can understand this difference and separate a task that you know the LLM can handle because you’ll be able to leverage its knowledge and judge the final work yourself.

But this distinction needs to be more clear, I think. The LLM doesn’t have any deep understanding of anything you’re creaming into the context window the same way it does with concepts it learned during training from millions of examples and building some internal model of the concepts and their relationships.