[throwawaymaths]

Sorry, how could there not be a valid next token? Presumably your interface would generate a state machine with appropriate masking arrays, and iirc generally speaking all 256 byte choices are in the token list. There's no way to get stuck in a place where the JSON is invalid? Can you give an example?

If you want to be really clever about your picker, a deterministic result would blat out the all the known possible strings.

For example, if you had an object with defined a defined set of properties, you could just go ahead and not bother generating tokens for all the properties and just tokenize, E.G. `{"foo":"` (6-ish tokens) without even passing through the LLM. As soon as an unescaped `"` arrives, you know the continuation must be `,"bar":"`, for example

> This greedy algorithm doesn't result in a qualitatively different distribution from a rejection sampling algorithm.

It absolutely will. But so will adding an extra newline in your prompt, for example. That sort of thing is part and parcel of how llms work

[contravariant]

Hmm, I think any example where it can get stuck is going to be a bit contrived since really it's a question of how easy it is to recognize a valid prefix. Say for example you want the LLM to generate a valid chess match and it ends up in a situation with just 2 kings left. If you're not careful with your definitions you could end up in an endless loop that never ends.

That said if you know all valid prefixes in your language in advance then you can always realise when a token leaves no valid continuations.

> It absolutely will. But so will adding an extra newline

A newline is less likely to dramatically drop the quality, a greedy method could easily end driving itself into a dead end (if not grammatically then semantically).

Say you want it to give a weather prediction consisting of a description followed by a tag 'sunny' or 'cloudy' and your model is on its way to generate

    { 
      desc: "Strong winds followed by heavy rainfall.", 
      tag: "stormy" 
    }

If it ever gets to the 's' in stormy it will be forced to pick 'sunny', even if that makes no sense in context.

[arjvik]

Schema needs to be a part of the prompt as well so it can associatively recall the options