Hacker News new | ask | show | jobs
by mewpmewp2 849 days ago
Where did you get that understanding? This doesn't really make any sense, how would GPT be able to stream token at a time in the first place?
1 comments

There's actually information provided during token generation that act as a level of confidence.

You can definitely stream and choose the highest scoring values amongst a few shots at generating the best next token candidate.