[amusedcyclist]

4 words a second doesn't seem fast enough for a voice assistant ?

[mromanuk]

It's faster than that [0], 20 token/s, should be approximately 15 words per second.

0: https://help.openai.com/en/articles/4936856-what-are-tokens-...

[schappim]

I've had difficulty obtaining useful results from the smaller (7B-sized) models. The issue lies in the content, not the speed. If you could stream the text-to-speech, the speed alone would be satisfactory.

[amusedcyclist]

You're right I overestimated how fast we talk!

[fwlr]

Some rules of thumb I use for estimating this kind of stuff

100wpm: Max typing speed

200wpm: Max speaking speed

300wpm: Max listening speed, max reading speed with subvocalisation

900wpm: Max reading speed without subvocalisation

[mromanuk]

Doin napkin math, this model should be hitting 900wpm