[gwern]

Most of the examples don't rhyme. It's unclear to me if this is because most of the original poetry doesn't rhyme so it's just faithfully replicating the lack of rhyme, or if it only partially and accidentally grasps the idea of rhyme.

As for the best one, I quote the ones that struck me during the training process, and some are highlighted in https://www.gwern.net/GPT-2#unconditional-samples

Some of the ones I like are 'We never say "Thank you"', 'Thy soul, thy very soul is burning!', '"It is morn!” said the clover-bush', 'And they have seen the last light fail', 'There comes a murmur low and sweet'.

Probably the best IMO is 'The sun is gone, and the night is late', but of course everyone will have a different favorite.

[p1esk]

Yes, "The sun is gone..." starts out amazingly well. But later fixates on tides for some reason :)

Everything is generated by the 117M model, correct? If so, do you expect the quality to improve for larger models, or is there not enough poetry to train them on? I wonder how much of total poetry is contained in Gutenberg poetry corpus...

By the way, here's some poetry which has been generated by a Markov model: http://www.kurzweilcyberart.com/poetry/rkcp_poetry_samples.p...

[gwern]

It's a mix of OA 117M and 345M at the moment. I haven't observed too much in the way of overfitting yet, so there should still be benefits to going up another 4.4x in model size to 1.5B. My guess is that at 1.5B, it'll start being more important to improve the poetry corpus, since you can already start to see problems with it - the Alexander Pope brokenness and the occasional prose generation of footnotes/commentary are definitely undesirable, and I suspect there would be less 'run on' effect in samples if the original corpus actually properly marked '<|endoftext|>' for each poem...