[lsy]

Is there some way of holding the LLM response to a given prompt constant? It sounds like a lot of this relies on the LLM getting the right answer in sequence, so I'm guessing they do something like keep the temperature at 0? Otherwise you are going to wind up with possibly different behavior run-to-run. And even if they do have something like the above, don't we end up with potentially breaking changes once models are updated? Basically the issue is that even if you can guarantee response format X for prompt A, a slightly modified prompt A' has no guarantee that its response will be in the same format as X, even in the same model. You can also imagine that the more "Tools" are available, the lower the chance that the model will pick the right one based on its English text description. Would be interesting to know how this is being addressed.

[newhouseb]

There are a couple different approaches:

- Rerun the prompt until you get a format that is consistent

- Steer the output token selection towards a predefined prompt

For the latter, I've built a proof of concept that takes in a JSON schema with a huggingface transformer and constrains token selection by modifying the output probabilities such that only schematically valid tokens can be emitted, see "Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema" @ https://github.com/newhouseb/clownfish. Unfortunately, given OpenAI's current API this is only possible on locally run models. That is... at any level of cost effectiveness. It's technically possible but (worst case) quadratically expensive against their current APIs.

[computerex]

The output is stochastic, so if the response doesn't decode to your format you can try the request again. Gpt3.5-turbo is pretty good at tool selection and use, but it often messes up with difficult tasks. gpt-4 is on another level when it comes to tool use. It is very reliable in my testing.

You ofc can't guarantee the output so defensive programming, retries are a must in my opinion. We are all learning how to work with this technology.

[Ozzie_osman]

Even with temperature 0 and the same model, the models are slightly non-deterministic and may diverge with the same input.

You probably don't want to treat them as deterministic (at least, not now). But there are many applications where slightly non-deterministic behavior is OK.

[jasonjmcghee]

I've been seeing the trick of creating a lookup table of query to response as a hack to "solve" this problem, with the other benefit of saving a call to the model. Especially useful when developing agents

[amrrs]

There is a library called guardrails . I've not played with it extensively but that seems to address issues like this one.

[Cyphase]

https://shreyar.github.io/guardrails/

https://github.com/shreyar/guardrails