[ekojs]

While the math seems intimidating, it does not look all too different from SPIN and previous researches. Pretty surprising how effective this is though. The costs here seems to be way higher too (with all the GPT4 calls).

> We also do a brief cost analysis associated with the scaled-up experiment on 600k training inputs. The major line items are the cost of sampling outputs, annotating them with GPT-4 to construct training pairs, and then training the next iteration against those pairs. For _each_ of the six iterations:

> 1. Sampling: it took about 18-24 hours to inference 5 outputs for all 100k examples on 10 8xA100 80GB pods, depending on the average length, costing about $6,000 based on spot pricing.

> 2. Annotation: the average number of prompt tokens sent to GPT-4 for annotation across iterations was about 450M, with an average of about 60M completion tokens, amounting to about $34,000 based on the version of the endpoint we were using.

> 3. Training: ironically, training was the cheapest step, taking only 12-24 hours on two 8xA100 80GB nodes

[visarga]

Yeah most of the cost in the future will be in preparing the training data by doing inference. This is the only way models can learn from their mistakes.