[Abishek_Muthian]

I'm more interested in running distributed inference for purpose built small language models than these coding LLMs.

Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.

Nicely done OP.

[unrvl22]

Something like this is nice, where instead of having 1 model with X active experts, you have 10 different models, all small and dense, trained on specific information. and loaded on 10 different servers, with one router.

[sig_kill]

This is actually how I develop and use the mesh at home. Rather than splitting models, I aggregate disparate compute behind one endpoint, without having separate inference providers on each host and a gateway like LiteLLM

[Abishek_Muthian]

True, I've been telling that coding LLMs have been detrimental to local AI scene. What happened to those nifty ML projects which can run on raspberry pi zero.