[jncfhnb]

… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?

[esafak]

If the LLM doesn't output data that conforms to a schema, you can't reliably parse it, so you're back to square one.

[jncfhnb]

It’s significantly easier to output an integer than a JSON with a key value structure where the value is an integer and everything else is exactly as desired

[esafak]

That's because you've dumbed down the problem. If it was just about outputting one integer, there would be nothing to discuss. Now add a bunch more fields, add some nesting and other constraints into it...

[jncfhnb]

The more complexity you add the less likely the LLM is to give you a valid response in one shot. It’s still going to be easier to get the LLM to supply values to a fixed scheme than to get the LLM to give the answers and the scheme

[neverokay]

Is there a general model that got fine tuned on these json schema/output pairs?

Seems like it would be universally useful.

[janpieterz]

How would I do this reliably? Eg give me 10 different values, all in one prompt for performance reasons?

Might not need JSON but whatever format it outputs, it needs to be reliable.

[jncfhnb]

Don’t do it all in one prompt.

[janpieterz]

Right, but now I’m basically running a huge performance hit, need to parallelize my queries etc.

I was parsing a document recently, 10-ish questions for 1 document, would make things expensive.

Might be what’s needed but not ideal.

[jncfhnb]

LLM performance is a function of the number of tokens, not queries