[317070]

there exists a setting which will work for all your test points.

The caveat of the author is (I think) that if you have a task, you collect a set of points (questions) on which you will test this task. Then you tune your setting (prompt) to start working for your test point (questions).

After that procedure, you do not know if that prompt solves the original task. You might have overfitted to your test points.

And by repeatedly doing this overfitting for various tasks, you are not gathering evidence that a good setting truly exists for all tasks

[loa_in_]

You can go as far as claiming that it's true for your colleagues as well! They've solved their piece of the job so far, but what evidence do you have that they will keep solving them in future? It's all just speculation!