[namaria]

I agree, it is absolutely not a skill. LLMs are a black box and the models keep changing under you, and their output can change if you try the exact same input more than once.

People claiming it's a skill should read up on experiments on behavior adaptation to stochastic rewards. Subjects develop elaborate "rain dances" in the belief that they can influence the outcome. Not unlike sports fans superstitions.

[fhd2]

This. If there was some stability in the space, you could empirically develop good practices that probably beat naive practices. But since everything changes every couple of months and since you'll usually want to try different models on an ongoing basis, I found I'm doing just fine with a very small bag of tricks.

Sure, by definition, prompting is a skill. But it's a skill that really isn't hard to learn, and the gap between a beginner and a master is pretty narrow. The real differentiator is understanding the domain you're promoting for deeply, e.g. software development or visual design. Most value comes out of knowing what to ask for, and knowing how to evaluate the results.

[or_am_i]

Analogy would have been correct if prompting didn't influence the output (which I hope you agree is not the case).

And yes, the model keeps changing under you -- much like a horse is changing under a jockey, forcing them to adapt. Or like formula drivers and different car brands.

You can absolutely improve the results by experimenting with prompting, by building a mental mode of what happens inside the "black box", by learning what kinds of context it has/does not have, how (not) to overburden it with instructions etc. etc.

[Xmd5a]

And yet prompts can be optimized.

[AlexeyBrin]

You can optimize a prompt for a particular LLM model and this can be done only through experimentation. If you take your heavily optimized prompt and apply it to a different model there is a good chance you need to start from scratch.

What you need to do every few months/weeks depending of when the last model was released is to reevaluate your bag of tricks.

At some point it becomes a roulette - you try this, you tray that and maybe it works or maybe not ...

[Xmd5a]

Stumbled upon this in another thread:

https://ai-analytics.wharton.upenn.edu/generative-ai-labs/re...

My point still holds that it is optimizable though (https://github.com/zou-group/textgrad, https://arxiv.org/abs/2501.16673)

>Subjects develop elaborate "rain dances" in the belief that they can influence the outcome. Not unlike sports fans superstitions.

Anybody tuning neural weights by hand would feel like this.