Hacker News new | ask | show | jobs
by meghan_rain 1202 days ago
how long for one token to infer on an average cpu?
2 comments
markasoftware 1202 days ago
I tested on a decidedly above average CPU, and got several words per second on the 7B model. I'd guess maybe one word per second on a more average one?
link
raverbashing 1202 days ago
Cool so we're back to the days of 2400 baud modems
link
Tepix 1202 days ago
More like 300 baud. At 300 baud (30 cps) you can still read it as it arrives.
link
asplake 1202 days ago
Simulating a slow typist
link
kristianp 1202 days ago
From the readme: On a Ryzen 7900X, the 7B model is able to infer several words per second, quite a lot better than you'd expect!
link