[bogwog]

Any idea how much GPU memory you'd need to run this locally?

EDIT: just tried it and it didn't seem to go past ~6gb

[lossolo]

It's 1 billion model with Fp16 precision so 4-6 GB max.