[xaxis]

I've started work on such a "Distributed Client-Side LLM" project but I'm skeptical of the practical use cases. Largely due to the inference that can be accomplished expediently across most client devices. I'm working on a SIMD (WASM compiled), WebGL, and WebGPU based inference engine for a couple of baseline models (llama-7b, etc).

So basically like WebLLM but that supports requests between user nodes. So entirely web based with a p2p orchestration layer serving inference requests across the connected nodes of the network, serving from active nodes that are not being used.

Anyhow, writing a cross-model highly-compatible inference engine that can fallback to minim available compute option (SIMD at the worst) is proving challenging enough, so I have lots of time to over think whether such a system would even prove useful.

A long while back (2011 - I'm old) I wrote a barely functional "Web CDN" that used WebRTC (the spec was very new) that distributed requests for content to a given site across active clients. This distributed LLM project is basically just that except instead of content (text, images, video) we're dealing with inference requests to supported models.