[jdlshore]

Carson’s experience matches mine: AI is good at analysis and boilerplate, but not good at the kind of critical thinking necessary for good designs. If it were human, I would say that it jumps to solutions to quickly, rather than stepping back to consider the big picture and how everything should fit together to make a cohesive whole.

It’s not human, of course, and I think this problem actually relates to the fact that LLMs don’t have a world model. They don’t study and think through a design in the way that humans do. They don’t form a mental model of how everything fits together and how that design can be tweaked to most elegantly support a change.

I suspect that this is a fundamental limitation of LLMs, and that design will remain a weak point until some sort of bespoke design AI is bolted onto the side. In the meantime, we’ve got a lot of people producing a lot of code very quickly, and I think the debt in that code is going to be a millstone around our necks for a long time to come.

[stymaar]

> I suspect that this is a fundamental limitation of LLMs

I suspect there's also a strong sociological bias at play: LLMs are being made by people who are familiar with coding but aren't software engineers. So they design their RL policies around the idea that the LLM must learn how to code, not that they must learn to design a maintenable piece of software.

[oulipo2]

Exactly, LLM is good at "code inpainting" : define clear structures and goals, and it will fill the boilerplate. But it doesn't work for reasoning and abstraction, so it fails to synthesise and propose novel views. But that's integral to the way it's designed and has been trained, to do a kind of "averaging" which limits it's capacity to explore novel designs

[thunky]

> But it doesn't work for reasoning and abstraction, so it fails to synthesise and propose novel views

I disagree. Have a conversation with it about your problem and work through design decisions with it. When I do that, I find it gives me a lot of good ideas.

Disclaimer: I'm not working on anything groundbreaking (like most people)

[sublinear]

Yes, but "good ideas" compared to what? If you were aware of the better alternatives, you probably wouldn't be discussing those details with an LLM. You'd find that it just randomly gave you one. It might work, but you don't know how well until you're already entrenched.

Nobody knows everything, so of course LLMs can be useful sometimes. More useful than plain old search, books, or even discussion with real humans? Maybe.

Search can offer a much broader context than an LLM hyperfocused on just generating text. Books may lead you to realize you were asking the wrong questions. Discussions will provide an overall "vibe" of the topic.

These are not competing options. We can and should be using all of them when possible.

[thunky]

> Yes, but "good ideas" compared to what? If you were aware of the better alternatives, you probably wouldn't be discussing those details with an LLM

Even when I already have a good idea of how I plan to do something, I may still ask AI and then find it gave me better idea for some particular thing.

I liken it to using GPS even when you know the route like the back of your hand. It can still steer you around an accident.

To do this effectively I have to drop the idea that I always know better than it does.

[sublinear]

If we are going to use the GPS analogy, not every trip is a route or even paved. GPS is notoriously unreliable when on a hike. You have to be a lot more careful than that.

Most real world software tends to be exactly that kind of situation. You need the rest of the business to help decide every detail of a service that will be in production long term. You are not just on a hike. Business is conquest. You are setting up camp with the lofty goal of scaling to a settlement, then a town, etc.

This sounds dramatic, but I think many are too easily impressed/jaded. Some people can't believe it, but this is still very early days for software. We're barely at the point where, maybe, the layperson can just about build small trivial gadgets for themselves. Meanwhile, there are people out there sailing the seas and beyond.

[thunky]

Sure, if you're one of the few that is building something truly novel, I agree AI will not be as useful to you.

That's why I added the disclaimer above: I'm not working on anything groundbreaking (like most people).

And I disagree with your premise overall, because I think the overwhelming majority of software development is much more like driving on a paved highway than it is like hiking through unmarked forest. Which is exactly why AI works so well: it's trained on thousands of examples of very similar solutions to very similar problems.

All of the hard work has already been done by people before us. We have the luxury of sitting down in front of incredible hardware, operating systems, fully designed languages, optimizing compilers, IDEs to fill in the blanks for us, and now AI to write up entire programs for us - none of which we had anything to do with the creation of. All we need to do is hook things together and slap on a layer of paint.

[altmanaltman]

Sure but you can also google your problem and check what is industry standard/what is the correct way to do things (imo in less time than it takes to go through a conversation).

But the problem is that when you ask ai to solve a problem on its own, its default plan can suck. You can mitigate that by research and context but it doesn't mean the initial problem is solved. But even that requires skill and human judgement (both ai conversation research or traditional research) and a lot of people want to skip that entirely.

[appplication]

I find I don’t necessarily need or want AI to give me ideas, but I would agree having a conversational back and forth generally yields decent results.

I have found being Socratic in my questions, and trying to get the AI to arrive at my intended design via such conversations supplies the right level of context for properly solving the problem. It’s token intensive, without a doubt, but I find the result is the AI tends to be better equipped to handle the many micro decisions that need to be made along the way.

The contrast to this is I give it a detailed prompt where it then asks questions of me, which also generally works but I find the AI tends to not be as well equipped for decisions it needs to make mid implementation.

It’s not perfect, and maybe not even a good fit for some. I also never know what to think when people tell me their idiosyncratic ways of using AI. Ultimately I think the most effective way is whatever lets you translate the vision in your head into the end result.

[oulipo2]

When I say "novel ideas" it means something groundbreaking, indeed, not rehashing common "this is the best practice for a simple CRUD backend"

[jacobedawson]

I feel as though that world model strongly correlates with memory - the experience of having jumped to a conclusion early and full-steaming ahead, only to be bitten by constraints and problems later down the track.

Part of that is critical thinking and projecting forward / simulating potential issues, and part of that is that memory which in humans we probably would see as "wisdom".

I don't know if that's a fundamental limitation of LLMs, or, rather, that this can be solved moving forward with better memory systems, harnesses, and context windows.

[mewpmewp2]

Yeah, I think it's more so learning from experiences that didn't scale. E.g. I started out with Notepad, and wrote everything for the website I wanted to build in a single large massive php file. I of course don't do that anymore, but it was a step by step iterative progress to move to where I am now. Although I still miss how easy it was to see changes locally, deploy to prod quickly, and make hotfixes in prod. I sometimes think maybe I should go back to php.

[recroad]

Have to disagree with this as it's excellent at helping you wide and broad before converging. I suggest trying OpenSpec and use /ospx:explore to state your problem and go from there.

[pgwhalen]

These takes Arne necessarily incompatible. It can be a great tool for helping you do this kind of big picture design, but still need you as a guide and taste-maker to get to a good end result.

[epolanski]

In my experience harness can do wonders to improve this.

Instead of asking it to generically to analyze and do X, you can use brainstorming skills like those from superpowers [1].

This makes it approach the problem better and keeps you in the loop.

Another step is then to have it review its plans by another LLM acting doing adversarial review. I have a claude skill [2] that calls codex to do it, and they chat among each other.

It's a tremendous boost in design quality.

[1] https://github.com/obra/Superpowers

[2] https://gist.github.com/enricopolanski/6c5038a8e20cc4098cd99...

[Zababa]

I don't think this problem is related to the fact that they don't have a world model, or because they don't form a mental model of how everything fits together, or a fundamental limitation of LLMs. These claims are often meaningless, and the boring answer is usually something like "software architecture is harder to verify than code/maths so RLing on it is harder, and it's harder writing good evals/benchmarks for it".

[rst]

One partial mitigation is to ask it to use plan mode -- and then very carefully review the plan before allowing it to execute.

[jdlshore]

My experience with AI plans is that they’re a wall of text that’s very hard to extract meaning from. Combined with it not doing a good job to begin with, I don’t think plan+revise is a great use of time.

[weakfish]

I feel the same way. Maybe it’s the ADHD, maybe I’m just dumb, but I cannot parse well the giant walls they tend to produce.

[carljungslabtek]

It’s melting my brain to read them all day. Our merge request descriptions are a mile long and so dense with jargon that it’s very difficult to figure out the important part of the changes.

They turned the english language into enterprise java and my train of thought is now a series of NullPointerExceptions

[clates]

> They turned the english language into enterprise java

.... tell them not to do that if you don't like it?

"PR Descriptions must explain the entirety of the PR's contents in 300 characters or less and be written at no greater than a 600 lexile score. After writing the description, carefully review it's claims against the changeset diff if any staged changes are unable to be tracked back to a claim in the PR description, reject the creation and alert the user of the discrepancy offering solutions on how to remediate"

[fugaziboutit]

An LLM conversation is like handling clay. When I don't grok an answer I mold the LLM's approach to fit my level of mastery of the subject. It's one of the few interactions you can have in life where you can tell someone how to talk to you without considering how they feel about being ordered around.

[mjfisher]

That's interesting and actually the opposite of mine. I wonder if it's stack or methodology dependant? For reference I'm usually using cursor and opus4.6 and for a bigger piece of work:

- Start in ask mode - "I'm planning on doing X to achieve Y; are there any alternative approaches? What problems might I run into?"

- Chat for a bit and get the high level approach, switch to plan mode and ask for a nicely formatted plan

- What's kicked out is already in the rough shape of the discussion so far, so it's a case of following a nicely formatted doc through and highlighting sections of text and asking for clarification or changes

- Hitting "build" and then reviewing what's been done

For a new service I might spend an hour in ask/plan mode - but then it gets 95% of the build itself right first time.

Do you do the same with different results, or is there a different stack/methodology you go through?

[knollimar]

I get a lot of this in design docs every time I give it a negative constraint:

[Suboptimal choice]

And here's why it's not suboptimal -- you said X sucks and notto do X, but this choice is not technically X, it's just really similar and shares that sucky property.

[bob1029]

I've been in a lot of situations where I could step gpt5.x through a big refactor if I spoon feed it one type name at a time. If I let it try to do the whole thing at once it will refuse or get stuck in apply patch loops.

Planner / executor separation can make a huge difference in performance. LLMs are fantastic at coming up with a lot of elaborate narratives regarding what should be done. They are terrible about doing that prescribed work all at once. This impedance mismatch is best resolved with a simple role separation. Placing a shared collection of tasks between these roles is how you can decouple them. The executors need significantly more tokens than your planners to get the job done. It's probably in the range of 10-100x more for really complicated jobs with a lot of iterations through compiler feedback, sql provider errors, etc. This is why you can't do both things in the same context very well.

[saagarjha]

At that point I would rather just write the plan myself

[altmanaltman]

Okay but that means you already know the plan since you are qualified to review it. So why not just tell it the plan yourself (0-shot) vrs having it guess and you review multiple times (n-shot). Wouldn't the former be more effective everytime?

[vb-8448]

It's just because not enough people had this very specific problem before.

This article will be part of the next model training set, and probably it will be able to solve it despite not understanding anything about world or not studying or thinking.