[seemaze]

And here I am with 128GB Strix Halo longingly eyeing the Blackwell cards that spit tokens 10-20x the speed.

The question is ultimate shape of knowledge compression and bandwidth optimization at which we arrive I suppose.

[canyp]

If you haven't already, check/increase the GPU memory carve-out on your UEFI.

More details: https://rocm.docs.amd.com/en/docs-7.2.0/how-to/system-optimi...

[seemaze]

Currently utilizing 126GB GTT on a headless host

[electroglyph]

that link actually recommends not doing it from UEFI and doing it via software