[LarsDu88]

At 9000 tokens/s you could interleave a lot of requests so long as pre-fill is also fast. It really depends on how much you need to keep sessions open to take advantage of KV caching