[leobg]

Good point with the HTML in the training data.

I always found the “structured output feature” thing odd. LLMs will follow any structure present in the prompt. So with a few few-shot examples, you have always been able to make them do anything - return JSON, Python function calls, or any syntax you come up with that works well for your task (ie streaming, separating thinking tokens from output, etc.).

All of that comes just with a little bit of overhead for the developer: Giving the examples, writing the parsing logic, and a strategy for the rare cases where the parsing may fail (retry, default value, prompt adaptation).

Once you embrace this approach, switching models is trivial. You can evaluate models with heavy RLHF like OpenAI’s against open models, base models, tiny models. Very easy to save a lot of money and achieve incredible speeds.