Hacker News new | ask | show | jobs
by tingletech 16 days ago
I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
1 comments

You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.
the grammar support/json-schema applies to a whole response, but I was under the impression a similar logit guide rail can be applied when a tool call is opened.