[nvm0n2]

Training needs insane amounts of inter node bandwidth, to the extent that training clusters use specially built hardware for it. Decentralised training isn't physically possible anytime soon, maybe never.