[TuringNYC]

I'd love a way to do this locally -- pool all the PCs in our own office for in-office pools of compute. Any suggestions from anyone? We currently run ollama but manually manage the pools

[damezumari]

https://github.com/exo-explore/exo

[zozbot234]

If you set CPUSchedulingPolicy=idle Nice=19 IOSchedulingClass=idle in the ollama server configuration it should run in the background with lowest priority.

[utopiah]

Seems like so much more work than "just" paying for https://huggingface.co or whichever other neocloud who already did all the setup for you and just waits for your credit card per minute/seconds/token.

[TuringNYC]

It is much more work because for many workloads you have geographic ringfencing and cannot send it out to the cloud

[utopiah]

Doubt this kind of workloads would agree to send data then to a cloud of randos devices, precisely when cloud providers to certify they aren't looking at clients data (Customer-managed encryption keys, CMEK).

[TuringNYC]

>> Doubt this kind of workloads would agree to send data then to a cloud of randos devices,

Totally agree, which is why i said "I'd love a way to do this locally -- pool all the PCs in our own office for in-office pools of compute."