[rbanffy]

These little devices are extremely interesting. I have a side project I will one day get started - to place 32 SoCs (or fewer SoCs with more cores), connect them via PCB traces to an ethernet hub (I need to learn how to do that), and leave one or more "upstream" network ports for connecting multiple boards together. Each core would light up a red LED on the front-side of the board via 90-degree LED holders.

Then I'd pack 16 of them, and build a tiny Connection Machine cube.

Not sure what I'd use a cluster of 512 very puny servers for though... I guess it'd be for learning how to manage clusters with unreasonable numbers of nodes.

[rasz]

You can do that right now with random raspberry pi, a string of addressable leds and a tub of glue

[Lerc]

I have always wanted to make a n-cube machine. I prefer the rp2350 myself, and have been quite keen on the thought of what you can do with PIO <-> PIO between chips.

And yes the main goal is to figure out how to program the thing in a way that balances ease of use with performance.

I also like the idea of a PSRAM junction, so that every core gets a PSRAM, but neighbours can swap ownership.

I had wondered what happens to the wireless spectrum if you tried it with ESP32. 512 devices in a small space yelling at each other.

[goodpoint]

little?

[rbanffy]

Once I figure out how to join that many cores via ethernet on PCB, it's a little project. SMD soldering should not be that troublesome.

[throwup238]

What are you having trouble figuring out?

ESP32 RGMII (32x) -> PHY (32x RTL8211F) -> Slave switch (6x RTL8367) -> master switch (1x) -> magnetics (for the external port). You’ll probably want a better IC for the master switch so I can’t name one of the top of my head but this would be a relatively simple, if large, PCB.

The hard part I think is verifying that all the PHYs and switches will work correctly without magnetics on a board to board connection.

[rbanffy]

The SoC (I was thinking about the Octavo parts) has two ethernet interfaces. We attach the interfaces to the RTL8211 and then to the switch (or do we need RTL8211s there as well? Is there a switch chip that can operate directly with the SoC ethernet ports?

I am way out of my depth here. I actively avoided analog design in college, and it kind of shows. :-(

[throwup238]

There are switches that have MAC-to-MAC skipping the PHY but only on one or two ports. I don’t know of any switch ICs that have more than two so you need to bring your own RTL8211 PHY for the rest of the ports anyway. You can just connect the SoC’s PHY directly to the switch’s ports, skipping magnetics.

There isn’t much analog design here, except the high speed digital signals for which you can just follow some basic rules of thumb (and IC routing guidelines!). You need to length match the busses and correctly route the differential pairs between the PHY and switch. If you know how to use Altium (I assume Kicad has similar features) you can do pin and part swapping with some clever placement to route only a few of the busses as short as possible, then clone them as rooms until you’ve got the number of nodes you need. The last bit will be routing the master switch that connects the slave switches and the external port.

If you’ve never done it before I’d be realistic and try to hit 100mbit first, maybe gigabit interconnection between the switches. The routing you’ll need at those slower speeds will be a lot more forgiving.