[potatoman22]

Depending on what sort of constraints you need on your output, a custom token sampler, logit bias, or verifying it against a grammar could do the trick.