[verdverm]

The tool I've been using, llm-compressor, can quant models that do not fit in memory (use the sequential pipeline)

https://github.com/vllm-project/llm-compressor

my setup to help you on your way: https://github.com/verdverm/quantr

Though it seems these will not be needed as Poolside has published quants & dflash with their models.

[alfiedotwtf]

Nice! Do you know of any tools that do this for tensorrt models?

[verdverm]

Sorry, I don't, still newer to the quantizing side of things since many are already produced by others.