Hacker News new | ask | show | jobs
by anemll 410 days ago
Right.I was thinking about it, you still need batch refill, however, Apple Core ML tools were failing for attention activations quantization. Long context, pre-fill is still compute bound.