[IanCal]

> Note the model is trained on data generated by GPT-4.

Is it? I couldn't find that in the page, and can't easily access the links. The previous paper used 1B tokens from GPT-3.5

> It's probably orders of magnitude more expensive to generate the data at current API prices.

If you're generating a billion tokens, you might do better with dedicated instances, iirc they used to say if you were doing more than a few hundred million a month dedicated things were cheaper.

[alecco]

It's in the Phi-1.5 technical paper. For phi-2 they bumped the number of tokens to 1.4 T and for sure most of it is generated, like previous models.

[IanCal]

I might be missing it but I can't find where it says how the data was generated, it mostly refers back to the previous paper which started they used 3.5

I'd not be too surprised but I can't find anything in the technical report paper saying they're using 4 specifically.