[dannyw]

You absolutely don't need to 'retrain' to reduce your context window. In vLLM it is an inference parameter. Smaller context window, smaller KV, less RAM needed to serve the same volume of requests.

With a lot of architectures, you technically don't need to retrain to extend the context window either; e.g. RoPE scaling; but performance is typically crap.