On the topic of JSON output from these models, someone has added context free grammars to llama.cpp. This enforces that the output matches the grammar, effectively zeroing the probability of the next token not conforming to it.
It's genius, but it's also solving the "easy" problem of checking syntax. If you ask an LLM to generate some structured data representing something you describe (or a program that "does X") checking the result for valid syntax is just the first step. You then need to check for semantic validity; i.e., is it what you want?
Oh yes, but it's nice that this technique enforces the grammar during generation, one token at a time, rather than having to check after completing the query and rerun for adjustments.
You're right, but an LLM is already trying to make sense (i.e. predict well) within the constraints given. So if you constrain the syntax, it's trying to fill it with the correct semantics. Doesn't always manage it, but it's trying.
This is similar to the way where if you ask a question in a given language, it responds in that language. But it still follows the instructions (hidden prompt) that was given to it only in English.
I.e. an LLM is essentially about finding an intersection of requirements in order to predict output.
Simply asking Bard in the right formatting works fine. Admittedly the formatting is a bit annoying. Asking for no preamble doesn't work, asking for no leader doesn't work, asking for no additional description doesn't work.
> Me: Please give me a 2 word JSON dictionary without first saying "Sure, here is the JSON" or a similar phrase. It's very important to me that I only get the JSON and no other words or phrases.
> Bard: *
> JSON
> {
> "word1": "hello",
> "word2": "world"
> }
I just tried this prompt verbatim 8 times and it didn’t work once. did you do this with a new chat or did you do it after a chain of earlier prompts?
I was eventually able to get it to do it after a long chain of prompts, but completely unable to in one shot, which is especially important for this kind of use case
For a less dramatic strategy with LLMs that expose the tokenizer vocabulary, you can use context-free grammars to constrain the logits according to the parser so that the LLMs only generate valid next tokens for the language.[0]
Right, so OP wants to treat Bard as an API. I don't dispute the thought, though it does run counter to conversational models. I'm slightly anxious about all the rogue conversational text parsing to use LLM's as an API.
The only reason they’re having a problem is due to the way they’re requesting this. OpenAI’s GPT can follow schemas for responses and issue commands or API requests if you tell it how. I’m sure Bard can do the same with the right request structure.
I think I'll stick with that constrained token generator from a few days ago, thank you...
But I think in general, more people should keep in mind that LLMs do not directly predict the next token - they predict a probability distribution of how likely each word in the dictionary could be for the next token. Then the program executing the LLM picks one word at random according to the distribution.
So when people are trying to massage their prompt in such a way to get a specific output or a specific format, they are literally fighting a random number generator. This seems sort of silly to me.
I have the same problem with ChatGPT (3/4). It was working nice before when I'd ask "output only code, not explanation, no other text" and now it's stopped. Need to try threatening it :)
bard is insane. like wow. like what Google was to anything that came before it. whether it can show bare JSON or not plays down the insane power of the fact that this is an LLM with full access to the current internet. I just asked it who won the presidential elections this year at the (relatively obscure) university I went to and it gave me the exact correct % results from the top 5 candidates. seriously wow
experimenting with this myself, after telling it off multiple times for not doing it, and it profusely apologising and even literally saying it’s not within its functionality, I tried one last time and it did do it
I can’t get it to do it in one shot though, and I’m sceptical that any of the successful examples in this thread can either
https://twitter.com/GrantSlatton/status/1657559506069463040
https://github.com/grantslatton/llama.cpp/commit/007e26a99d4...
It's so obvious, it's genius.