[blindriver]

Conversely I think that 2x being the least amount of difference between mediocre and best prompts seems unbelievable. I've been using LLMs heavily just like everyone else, and I find that simple direct prompts get me most of what I need. I've never seen it get me less than half of what I expect at all.

[sokoloff]

Think of Google searching circa 2010 (before they completely screwed it up for profit). The best Google search user was way more effective than a mediocre searcher, even on the much simpler task of searching.

I don’t think you’re giving yourself enough credit for understanding enough about system design to be able to effectively prompt and guide the LLM.

John Carmack* is going to be at least twice as good at making a game with LLM coding assistants as a mediocre prompt engineer will be. Guido van Rossum* and Rich Hickey* will be over 2x a mediocre prompt engineer at language creation. Linus Torvalds* will create the first version of git far faster than any mediocre prompt engineer (who will never complete that task). And on and on…

* replace with whomever you think is “best” in the field you’re comparing.

[blindriver]

I guess I disagree with you. I think that results between average and the "best" prompters are getting closer and closer together because that's what LLM designers and customers want.

[sokoloff]

Sure. Everyone has incentive to raise the effectiveness of the bottom of the distribution as a buyer or seller of these tools. That’s where the big payday is, but incentive alignment doesn’t mean that it’s possible to lift mediocre all the way up to sit right next to best.