[accrual]

I wondered similar. Perhaps a local model cached in a 16GB or 24GB graphics card would perform well too. It would have to be a quantized/distilled model, but maybe sufficient, especially with some additional training as you mentioned.

[jszymborski]

If Qwen 0.6B is suitable, then it could fit in 576MB of VRAM[0].

https://huggingface.co/unsloth/Qwen3-0.6B-unsloth-bnb-4bit

[numpad0]

or on a single Axera AX630C module: https://www.youtube.com/watch?v=cMF6OfktIGg&t=25s

[otabdeveloper4]

16Gb is way overkill for this.