[tucnak]

>But it struggles with json

You should customise your sampler to mandate JSON grammar after ```json tokens.

[ed]

Grammar samplers are clever! But in the case of a missing escape character you’ll end up with a corrupted string.

Take for example: "A dog says \"Woof!\""

With a grammar, you’ll end up with "A dog says " when the model forgets to escape.

Which is valid JSON, but not what the model intended.

So it’s usually better to catch the exception and ask the model to try again.

Unless you’ve come across a sampler with backtracking? That would be cool