|
|
|
|
|
|
by cyanydeez
8 days ago
|
|
|
yes. I build a docker file from this: https://github.com/kyuz0/amd-strix-halo-toolboxes It has all the backends you want. Just a script rebuilds it. Then I just put it in a docker compose and whenever we want to update it, we pull the repo and rebuild it. I just a models.ini like:
version = 1
```
[*]
n-gpu-layers = 999
threads = 28
flash-attn = on
mlock = off
mmap = off
fit = off
warmup = on
jinja = true
direct-io = on
cache-prompt = true
cache-reuse = 256
models-max = 1
models-autoload = 1
timeout = 600
ctx-checkpoints = 256
cache-ram = 0
mlock = on
kv-unified = on
no-context-shift = on
swa-full = on [Q36-35B-A3B]
model = Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf
mmproj = mmproj-BF16.gguf
rope-scaling = yarn
rope-scale = 1.14441
yarn-orig-ctx = 262144
override-kv = qwen35.context_length=int:1000000
ctx-size = 300000
reasoning = auto
reasoning-budget = 421
reasoning-budget-message = you're reasoning too much; compress context and use subagents to investigate further
spec-type = draft-mtp
spec-draft-n-max = 3
batch-size = 16001
ubatch-size = 16001
cache-idle-slots = on
temp = 0.6
top-p = 0.95
min-p = 0.00
top-k = 20
repeat-penalty = 1.1
presence-penalty = 0.0
``` Using subagents, you keep the context size from bloating, and add tooling for the context compaction and it can keep looping. You probably want to clean up opencode's agent; back in the "Prompt Engineering" days there's a lot of unnecessary fluff. The goal of a prompt should be to get the model in the correct starting place, and not, try to give it everything it ever needs to do what you want it to do. |
|
|