[happyPersonR]

Pretty sure llama.cpp can already do that

[TYMorningCoffee]

I forgot to clarify dealing with the network bottleneck

[moralestapia]

Just my two cents from experience, any sufficiently advanced LLM training or inference pipeline eventually figures out that the real bottleneck is the network!