[rbanffy]

The SoC (I was thinking about the Octavo parts) has two ethernet interfaces. We attach the interfaces to the RTL8211 and then to the switch (or do we need RTL8211s there as well? Is there a switch chip that can operate directly with the SoC ethernet ports?

I am way out of my depth here. I actively avoided analog design in college, and it kind of shows. :-(

[throwup238]

There are switches that have MAC-to-MAC skipping the PHY but only on one or two ports. I don’t know of any switch ICs that have more than two so you need to bring your own RTL8211 PHY for the rest of the ports anyway. You can just connect the SoC’s PHY directly to the switch’s ports, skipping magnetics.

There isn’t much analog design here, except the high speed digital signals for which you can just follow some basic rules of thumb (and IC routing guidelines!). You need to length match the busses and correctly route the differential pairs between the PHY and switch. If you know how to use Altium (I assume Kicad has similar features) you can do pin and part swapping with some clever placement to route only a few of the busses as short as possible, then clone them as rooms until you’ve got the number of nodes you need. The last bit will be routing the master switch that connects the slave switches and the external port.

If you’ve never done it before I’d be realistic and try to hit 100mbit first, maybe gigabit interconnection between the switches. The routing you’ll need at those slower speeds will be a lot more forgiving.