Depending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space.
I can see that being cheaper to bitbang with PIO than to actually compute.
There's certainly some latency stack up, but throughput should be remarkably good.
I can see that being cheaper to bitbang with PIO than to actually compute.
There's certainly some latency stack up, but throughput should be remarkably good.