[mryab]

There is! See https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)

[exebook]

Amazing work! If I had a GPU, I'd join. I know similar project for text-to-image: https://aqualxx.github.io/stable-ui/

[efreak]

https://github.com/aqualxx/stable-horde-notebook

My only problem with stable horde is that their anti-cp measure involves checking the prompt for words like small, meaning I can't use a nsfw-capable model with certain prompts (holding a very small bag, etc). That, and seeing great things in the image rating and being unable to reproduce because it doesn't provide the prompt.