Not my field, but from this[1] blog post which references this[2] paper, it would seem so. Note the optimal approach are a bit different between training and inference. Also note that several of the approaches rely on batching multiple requests (prompts) in order to exploit the parallelism, so won't see the same gains if fed only a single prompt at a time.
[1]: https://medium.com/@plienhar/llm-inference-series-4-kv-cachi...
[2]: https://arxiv.org/abs/2104.04473