[mikewarot]

If I understand how Large Language models work, they don't actually know about spelling.... they are given tokens that represent words, and can only infer things from the context of those tokens across terabytes of data that they're given.

Any rhyming done is an impressive result.

[ben0x539]

I mean, English spelling certainly doesn't free you from the need to figure rhyming out from context.

[ramblerman]

It is amazing, but somewhat explicable as an emergent effect.

Dont forget that the model has seen all the poems and raps on the internet. It built some latent space where certain words always cluster together in the context of poems, and in which location.

In this case it really has the best database available to say, what next word would slot in nicely here - as that is precisely what it was trained to do.

It is amazing, but somewhat explicable as an emergent effect.

Dont forget that the model has seen all the poems and raps on the internet. It built some latent space where certain words always cluster together in the context of poems, and in which location.

In this case it really has the best database available to say, what next word would slot in nicely here - as that is precisely how it was trained.

I find it more amazing tbh that you can ask for a poem about something, and that it then sticks to the plot, makes references to the start etc than the actual rhyming.

[chewxy]

Well the nature of byte pair encoding has a side effect of encoding sounds as well. I briefly explored it here: https://youtu.be/fTDQUxha9qU?t=480