[thuttinger]

Works great! However, i had Python 3.11 set up as default python3 in path, and since there is no wheel for torch for 3.11 yet, the script failed. With 3.10 it worked flawlessly.

Small improvement: the node script could check if the model files are already present at the download location and not download them again in this case.

[hbbio]

Happened to me as well. Apparently, you can just run:

   python3.10 convert-pth-to-ggml.py models/7B 1
   ./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2

And then play with:

    ./main -m ./models/7B/ggml-model-q4_0.bin -t 8 -n 128 -p "..."