Hacker News new | ask | show | jobs
by rao-v 6 days ago
This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
1 comments
Lerc 6 days ago
One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions.

PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time.

The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.

link
NuclearPM 6 days ago
Run the numbers before you waste time here. I doubt this will work.
link
Lerc 6 days ago
PIO to PIO between two rp2350s should be able to transfer as many bits per clock as you can spare pins for.

They have a single cycle double multiply per core, and the interpolators give you a heap of ability

The PIO can be awkward, but you can run a bunch of them at once. Going from MCU to MCU you don't even need to involve the CPU cores, PIO to PIO Comms via pins

You are obviously not going to get big TOPS from it because a Trillion is a ridiculous amount anyway. But never underestimate the power of controlling the whole pipeline.

Ultimately none of the other things I'm doing with MCUs are practical, why would this to be any different.

link
JK-Swizzle 6 days ago
Running some quick numbers shows you should be able to get >1Gbps. But I seriously doubt you could get those speeds in reality. You would need to get them perfectly in sync, which would likely take a dedicated board and some great knowledge of the oscillator.

As someone who has done a reasonable amount with PIO, I do not think this is possible. However, that should not stop you. If you get it to work, please ping me.

link
hgoel 6 days ago
Wouldn't a layer per MCU be heavily bandwidth constrained?
link
monocasa 6 days ago
Depending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space.

I can see that being cheaper to bitbang with PIO than to actually compute.

There's certainly some latency stack up, but throughput should be remarkably good.

link