[newhouseb]

There are a couple different approaches:

- Rerun the prompt until you get a format that is consistent

- Steer the output token selection towards a predefined prompt

For the latter, I've built a proof of concept that takes in a JSON schema with a huggingface transformer and constrains token selection by modifying the output probabilities such that only schematically valid tokens can be emitted, see "Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema" @ https://github.com/newhouseb/clownfish. Unfortunately, given OpenAI's current API this is only possible on locally run models. That is... at any level of cost effectiveness. It's technically possible but (worst case) quadratically expensive against their current APIs.