[gwerbin]

> I am not sure how you imagine it being significantly improved, from a user point of view, without some kind of paradigm shift

I proposed how. New harness techniques and new training data/techniques, so the harness gets better and the LLM can be trained to work better with the harness. There's no reason to believe we're out of momentum for improvement in that direction.

[js8]

Yeah but what do you mean by (substantially) better in this context, what is the outcome? Modern models can understand the requirements as well as humans can.

However, they also make mistakes like humans, I don't think a better harness or better training will fix that, because fundamentally, they cannot read your mind, if you put in an ambiguous prompt.

I like to compare the process of turning inexact text to formal language to an error-correcting code. If you haven't made too much mistakes or have been precise in the specification, it will self-correct and do what you want. But if your input is too ambiguous, it will never do exactly what you want, but something close to it. And people (who are using AI) are still learning where is the boundary and how to tell.

The companies building these models are training them to react to typical expectations. If you have some special need, you will always have to tell the model, otherwise it will not know your exact context. And the harnesses have many tools for that or try to do that automatically already.