Does a system exist where we can train open source LLM models in this way? Maybe distribute it across a cluster of computers with credit points that can be later used for inference.
Not for training, but execution in a SheepIt-similar way can be done with AI Horde, which not only does LLM inference but also Stable Diffusion. https://stablehorde.net/