[tarpitt]

I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2

I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.

>SSD Wear Warning

> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.

Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?

[vforno]

I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!

[tarpitt]

I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.

I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.

If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.

I'm going to benchmark this right now with what I have and I'll get back to you on github.

[halcyonblue]

At least for NVME, it is the write cycles that are limited. Read cycles are non-destructive and essentially unlimited.

[throwdbaaway]

If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.

[vforno]

Really thanks!!

[valicord]

> OS page cache can generate writes

Is this a hallucination? What am I missing? Why would heavy reads generate writes?

[TacticalCoder]

> Is this a hallucination? What am I missing? Why would heavy reads generate writes?

I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?

I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?

[fallingbananna]

Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.

[onlyrealcuzzo]

Spilling

[gcr]

Wouldn’t turning off swap fix this issue?

[throwa356262]

Better to just change swapiness?

https://askubuntu.com/questions/103915/how-do-i-configure-sw...

[vforno]

That's possibly a good idea! We can work on it!

[tarpitt]

I also just edited my comment with more ideas in the beginning, sorry