[joatmon-snoo]

Author here- besides hellovai’s point about the performance bottleneck, it’s a really tricky semantic problem!

LLMs today are really good at producing output that satisfies the very vague metric of “this looks good to a human” but aren’t nearly as good at producing output that satisfies a complex set of syntax and schema constraints. The state space of the former is much larger than the latter, so there’s a lot more opportunity for an LLM to be successful by targeting the state space of “looks good to a human”. Plus, there’s still a lot of room for advancement in multimodality and data quality improvements.

Search problems, in general, deal with this too: it’s easy to provide a good search experience when there are a lot of high-quality candidates, and much harder when there are fewer, because all you have to do is return just a few of the best candidates. (This is partly why Google Drive Search has always sucked compared to Web Search- it’s really hard to guess exactly which document in a 10k-file-Drive a user is looking for, as opposed to finding something on Wikipedia/NYTimes/Instagram that the user might be looking for!)