[fxtentacle]

In general, that should work. Here's an example board with 4x CM4 modules and a RTL8367N switching chip: https://github.com/will127534/Miniature-CM4-Cluster

[azonenberg]

You're going to at least want to AC couple through some capacitors to avoid problems with common mode / RX bias issues between the different PHYs. But if it's on-mobo you don't need to have as much of a fault voltage rating compared to standard Ethernet which IIRC specs 1 kV isolation on the transformers.