The motherboard has 7 PCie slots and there are 8 GPUs. So where does the spare one connect to? Is he using two GPUs in the same slot limiting the bandwidth?
It’s an epyc server board, it probably has actual U.2/MCIO pcie ports on the board that can be merged back into a 16x slot in the bios. I had/have several boards like that.