[willkc]

I wish I'd had this tech, when I started long ago. I'm curious: is your actionable feedback always going to give an answer / correct the code?

E.g. What's the line between prompting GPT3 (assumption) to correct the code vs. explain the error vs. explain how to think about the problem.

[mirakyanmartin]

We did a couple of tricks to get good suggestions from the model (although some of them were changed later):

1. We tried to generate several explanation + code samples with the model. After which the code was parsed and evaluated with the judging system. Finally, only the code that would yield a correct result after the evaluation would be shown to the user. This was actually modified in the latest release we made.

2. We prompt different types of responses - suggestions on how to modify the code, suggestions of completely new ideas, and suggestions on which parts of the code to fix exactly.