[tssge]

The GPU has INT4, INT8, BF16 and FP16. Notably no FP8 or FP4.The official GPTQ-Int4 release from Qwen is a great quant for this but custom kernels are still rare for this hardware.

[moffkalast]

Must be a case of the hardware being there and the software not actually supporting it then.