[shirman]

Hi, it does not work with llama.cpp right?

[codelion]

Optillm works with llama.cpp but this approach is implemented as a decoding strategy in PyTorch so at the moment you will need to use the local inference server in optillm to use it.