[phillipcarter]

Unfortunately, this is extremely hard to do for two reasons:

1. The input space is boundless. Any natural language input, with any optional source of data, for any arbitrary use case is what's possible. But that means it's awfully hard to tell if a response can "improve" or not in advance without applying it to your use case.

2. The output space is so hard to measure! Usefulness can also mean different things to different people, especially once you get out of "better search engine" use cases and actually use GPT to produce a creative output.

[visarga]

everyone is using LLM as a judge to fix unbound output eval

[phillipcarter]

No, not everyone is. It's one of the ways you can do this, though.

[visarga]

Maybe I exaggerated a bit, but there are many papers today going this route.