[BoorishBears]

I really don't get what people are doing wrong here.

I have a 7000 token prompt that generates JSON chugging away in production and at scale I'm seeing ~1 in 4000 generations require a re-generation, and even that could probably be killed with some basic "healing" code.

OSS are prone to outputting garbage in my experience, but OP mentions ChatGPT:

How are you running into issues if you simply prefill the response with ```json and set ``` as your stop token?

Also, are people also just not trying to parse the opening and closing bracket and treating it as broken if there's a preamble? The prefill gets rid of the preamble, but if you're not willing/able to prefill, how hard is getting JSON out of a string?