[mycall]

Maybe the training approaches taken to date are wrong for decentralized systems. Setup a virtual subnet you can trust and do training on that. Create a AI model island in a trusted/federated model system -- definitely slower than the typical 'one big model' approach, but scalable to world size modeling.

Also, it wouldn't be able to use a transformer architecture. For inspiration, take a look at Google Maps and how it a much more efficient A* divide/conquer hill-climbing architecture. Think minimized matrix math.

[bradfa]

Other comments also hint at this idea, a distributed training solution is currently an open research problem. Solving it is not easy, yet. But 10 years ago what we have today for LLMs would have looked similarly impossible, so have hope, and apply yourself to the problem if you find it interesting!