3. make that output useful to my non-english-speaking software by structuring it.
Forcing step three to happen in step 2 means you're in a very unusual/unrealistic "head space"/context for next word completion.
Leaving structuring as a later step means the human like output/reasoning can be maintained for step 2, with the inhuman like structuring being more of a reformatting than reasoning+reformatting.
Leaving the "context" as small and precise as possible (which might me large for a given task) gets the best output, from what I've seen. Being a next word predictor, this makes sense to me.
I suppose this is just "chain of thought".
This is pretty easy to benchmark. Have it put the answer into json format, after producing the answer. Then, have it come up with an answer in json format, as one step. It's a night and day difference, at least for my tasks.
1. take some input
2. run it through the llm to get some output
3. make that output useful to my non-english-speaking software by structuring it.
Forcing step three to happen in step 2 means you're in a very unusual/unrealistic "head space"/context for next word completion.
Leaving structuring as a later step means the human like output/reasoning can be maintained for step 2, with the inhuman like structuring being more of a reformatting than reasoning+reformatting.
Leaving the "context" as small and precise as possible (which might me large for a given task) gets the best output, from what I've seen. Being a next word predictor, this makes sense to me.
I suppose this is just "chain of thought".
This is pretty easy to benchmark. Have it put the answer into json format, after producing the answer. Then, have it come up with an answer in json format, as one step. It's a night and day difference, at least for my tasks.