[visarga]

I was hoping to find some insights about why performance drops when using actual structured outputs. It's been a known problem. For example this paper "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" says:

> Surprisingly, we observe a significant decline in LLMs’ reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.

https://arxiv.org/abs/2408.02442v1

[roadside_picnic]

That paper had some serious methodological issues and the results have been shown to be misunderstood/incorrect in the majority of cases. In fact, in many cases structured outputs have shown to improve the quality of the results from an LLM (at least in terms of evaluation performance). The team at behind the Outlines library released a response the covers the issues in details and provides more information about structured outputs [0].

0. https://blog.dottxt.ai/say-what-you-mean.html