Hacker News new | ask | show | jobs
by scrlk 5 hours ago
IME, unquantised -> FP8 is pretty much lossless. What matters more is having an unquantized KV cache - using an FP8 KV cache can result in a significant drop in quality.
2 comments

>unquantised -> FP8 is pretty much lossless

Claude Shannon is rolling in his grave.

Do infra providers reveal that level of implementation detail?
I've seen a few articles from providers talking about KV cache quantisation, but it's not something they explicitly point out like they do with weights.

So you could end up paying more for unquantised weights, only to get silently hit with a quantised KV cache...