[bagels]

How great is the performance? Tokens/s?

[yjftsjthsd-h]

Random sample query ("What shape should a kumquat be?") against a 7B model quantised to 4b running on an i7-9750H (so a good CPU, but also a good laptop CPU from 2019) gives:

  148 tokens predicted, 159 ms per token, 6.27 tokens per second

[bagels]

Thanks, that helps.