I am a Vicuna developer. We plan to release the weights once we have addressed all concerns and have a low-resource version of the inference code ready. We released the demo first to get some early feedback on the model.
We hear a lot about "concerns" and many of us don't share the same ones... It would be good for clarity to know what are the concerns you feel are important enough to hold back releasing the weights?
It is mainly because of the legal issues caused by the license of llama model weights. We need to figure it out with Meta's llama team before releasing.
It would be great if you can help me with this PR as well as adding a support for exporting a model that was quantized using GPTQ, bitsandbytes, plain torch. This would bring a lot of benefit from both worlds:
- Low memory footprint(thanks quantization)
- Fast inference(thanks io binding)
Particularly in case of alpaca I have seen a 5x decrease in latency on A100 and 10x on AMD EPYC.
I believe this is the way for users to have an AI that could genereate a response as fast as it can on their hardware.
I have also added a link to my profile on hf with small alpacas turned into ONNX format. Take a look into them.
Has LoRA been considered as possible alternative for finetuning on your dataset? In that case releasing the 'diff' against the LLaMA weights would be simpler to work with.