[Muhtasham]

Thanks for detailed analysis, would be curios to see FP8 comparison too, given vllm has some custom kernels

[cheptsov]

Unfortunately, neither VLLM nor TGI support FP8 on AMD yet. But once they do, we will look into it.