|
|
|
|
|
|
by Aerroon
1033 days ago
|
|
|
You could just try it out if you have the hardware at home. Grab KoboldCPP and a GGML model from TheBloke that would fit your RAM/VRAM and try it. Make sure you follow the prompt structure for the model that you will see on TheBloke's download page for the model (very important). KoboldCPP: https://github.com/LostRuins/koboldcpp TheBloke: https://huggingface.co/TheBloke I would start with a 13b or 7b model quantized to 4-bits just to get the hang of it. Some generic or story telling model. Just make sure you follow the prompt structure that the model card lists. KoboldCPP is very easy to use. You just drag the model file onto the executable, wait till it loads and go to the web interface. |
|
|