[simonw]

This feels solvable to me. I wonder if you could use fine tuning against LLaMA to teach it to do this better?

GPT-3 etc can only do this because they had a LOT of code included in their training sets.

The LLaMA paper says Github was 4.5% of the training corpus, so maybe it does have that stuff baked in and just needs extra tuning or different prompts to tap into that knowledge.

[avereveard]

I have done it trough stages, so first stages emits in natural language in the format of "context: ... and question: ...." and then the second stage collect it as json, but then wait time doubles.