[herpderperator]

I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?

[StrauXX]

I'd expect that sampling would happen in software. Probably the hardware system would output a vector of probabilities over the tokenspace, just as the nets do when run in software.

I don't know though and am not aware of any docs going into detail here. That being said, sampling is really cheap. So implementing it in hardware wouldn't be worth it.