[ljosifov]

I usually doubt the 'small dataset tuned' variants. B/c ages ago (in the NN prehistory) I've done some NN training, and appreciate how hard it is to improve in general, and how easy it is to ruin a model in general while targeting a small dataset (LoRA-s are ok, that's different). That model/quant was the most recent one I was trying. But could not really use any of them, as the combo model + llama-server ground to a halt even at small context depth sizes on the amd gpu.

Yesterday I finally found a good combo! So writing this for the benefit for anyone that may have the same h/w. Got around to /GOAL search for something better for the h/w (amd 7900xtx), and pi agent found a new best that actually seems it will be useful for real. As the 40 tok/s speed starts dropping only at 260K context depth?? Served by hipfire from this repo https://github.com/Kaden-Schutt/hipfire, that worked the best got on llama-benchy:

  | Context | Wall Time | PP (t/s) | TG (t/s) |
  |:-------:|:---------:|:--------:|:--------:|
  |   1,024 |   17.3s.  |   660    |    40.8  |
  |   8,192 |   29.3s   |   606    |    40.8  |
  |  32,768 |   70.5s   |   599    |    40.7  |
  |  65,536 |  126.5s   |   594    |    40.6  |
  | 131,072 |  235.5s   |   591    |    39.0  |
  | 196,608 |  349.0s   |   591    |    40.2  |
  | 260,000 |  451.3s   |   594    |    37.2  |

This is - hipfire daemon dflash_mode auto, serve qwen3.6:27b --kv-mode asym2. Models:

  hipfire pull qwen3.6:27b           # 14 GB MQ4 quant, qwen3.6-27b.mq4
  hipfire pull qwen3.6:27b-draft     # 0.92 GB DFlash draft, qwen36-27b-dflash-mq4.hfq