[vidarh]

Because very few knows how to use AI. I teach AI courses on the side. I've done auditing supervised fine tuning and RLHF projects for a major provider. From seeing real prompts, many specifically from people who work with agents every day, people do not yet have the faintest clue how to productively prompt AI. A lot of people prompt them in ways that are barely coherent.

Even if models stopped improving today, it'd take years before we see the full effects of people slowly gaining the skills needed to leverage them.

[sarchertech]

Sure there are people holding it wrong.

But there are thousands of people on social media claiming huge productivity gains. Surely at least 5% of devs are holding it right.

If a 10x boost is possible, we’d notice that. There are only 20k games a year released on steam.

If my hypothesis is true and the real final output boost is somewhere near 20%, we’re seeing exactly what you’d expect.

[hackingonempty]

I'd love to look at what you consider to be good prompts if you could provide a link.

[vidarh]

You'd be surprised how low the bar is. What I'm seeing is down to the level of people not writing complete sentences.

There doesn't need to be any "magic" there. Just clearly state your requirements. And start by asking the model to plan out the changes and write a markdown file with a plan first (I prefer this over e.g. Claude Code's plan mode, because I like to keep that artefact), including planning out tests.

If a colleague of yours not intimately familiar with the project could get the plan without needing to ask followup questions (but able to spend time digging through the code), you've done pretty well.

You can go over-board with agents to assist in reviewing the code, running tests etc. as well, but that's the second 90%. The first 90% is just to write a coherent request for a plan, read the plan, ask for revisions until it makes sense, and tell it to implement it.

[nunez]

Not surprising. Many folks struggle with writing (hence why ChatGPT is so popular for writing stuff), so people struggling to coherently express what they want and how makes sense.

But the big models have come a long way in this regard. Claude + Opus especially. You can build something with a super small prompt and keep hammering it with fix prompts until you get what you want. It's not efficient, but it's doable, and it's much better than having to write a full spec not half a year ago.

[vidarh]

This is exactly it. A lot of people use it that way. And it's still a vast improvement, but they could also generally do a lot better with some training. I think this is one of the areas where you'll unfortunately see a big gap developing between developers who do this well, and have the models work undisturbed for longer and longer while doing other stuff, and those who ends up needing a lot more rework than necessary.

[KellyCriterion]

> Claude + Opus especially. You can build something with a super small prompt and keep hammering it with fix prompts until you get what you want.

LOL: especially with Claude this was only in 1 out of 10 cases?

Claude output is usually (near) production ready on the first prompt if you precisely describe where you are, what you want and how you get it and what the result should be.

[munch117]

> Just clearly state your requirements.

Nothing new here. Getting users to clearly state their requirements has always been like pulling teeth. Incomplete sentences and all.

If the people you are teaching are developers, they should know better. But I'm not all that surprised if many of them don't. People will be people.

[vidarh]

You're right, they should know better, but I think a lot of them have gotten away with it because most of them are not expected to produce written material setting out missing assumptions etc. and breaking down the task into more detail before proceeding to work, so a lot have never gotten the practice.

Once people have had the experience of being a lead and having to pass tasks to other developers a few times, most seem to develop this skill at least to a basic level, but even then it's often informal and they don't get enough practice documenting the details in one go, say by improving a ticket.

[gsnedders]

One thing that I’ve often seen is models, when very much told to just write a plan, still including sizeable amounts of code in the plan.

Maybe it’s needing to step back and even ask for design doc before a plan, but even then…