[sig_kill]

It depends on the size / structure of the model… but generally you can check out the MeshLLM huggingface org for pre-computed model splits.

We have a job that will take popular models, and dice them up and post on HF.

When that specific model is served, the layers will be shovelled around the available nodes on the mesh.

[throw1234567891]

Do you have instructions how to dice models myself? Every model in your org is a measly 4bit. I kinda hoped it is as simple as taking a GGUF and throwing at Mesh LLM but I fear it's not going to be that straightforward :)