[alxfoster]

FYI, many of us are indeed running 65B. I’m running 65B at 4-bit and getting about 7.5 tokens per second. Granted, I have a beefy machine with 2x 3090s and Nvlink but certainly well within the realm of any small lab.