Tim Dettmers amazing GPU blog post posits NVLink doesn't start to become useful until you are at 128+ GPUs
https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...