|
|
|
|
|
|
by xp84
999 days ago
|
|
|
Seriously disappointing. I was expecting that it would not produce total gibberish. It acts like it's a Markov chain, and only considers the last 1-2 words. Identical to the currently-shipping thing that we've had for the past however-many years. |
|
|
The default goal everyone is assuming is spitting out the longest correct sequence possible.
But in reality the mental cost of a wildly wrong prediction is much higher than the mental cost of a slightly wrong one, so what you'd train the model for is sequences of a few words at most being with higher confidence.
Most people can/will tune out slightly wrong words especially as they get a feel for what autocorrect is good and bad at.
If you unleash the full range of tokens GPT 2 can normally output, you'll constantly be blasting out words they didn't expect.
—
The fact your long sequence prediction got better doesn't matter because the UI is autocomplete not "auto-write": they're still expecting to drive, and a smart but noisy copilot is worse than a dumb but lazy one in that case.
I wouldn't be surprised if they trained the model to an effective context window of just a few hundred tokens with that in mind