[schmorptron]

LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting idle most of the time. Don't get me wrong, most desktop workloads are bursty, but the power needed to to them has gotten cheap enough that we can have way overkill for idle scenarios hardware just sitting on our desks.

[zozbot234]

Why does it have to be so bursty though? Just let it run multiple continuous-batched inferences overnight. This would work especially well in combination with SSD offload, and given any kind of sparse attention (common in more recent models) even swapping out the KV cache itself to disk might ultimately be a win. I wouldn't be surprised if something like that ultimately became feasible for single users running even K3 itself.

[bavell]

I find it difficult to always have one or more long-horizon tasks 'queued up' and ready to run... I find myself usually bottlenecked on design, review, or something similar that requires me being in the driver's seat. It's possible I could queue up a bunch of tasks, letting the LLM run off in multiple directions, but then I'd be less able to steer and course correct.

Just my experience though, I'm still figuring things out. Perhaps some subsets of tasks would be more ideal for these long-horizon workloads - exploration, multiple competing implementations, etc...

[trollbridge]

Use your favourite harness to help you find long-horizon tasks to have queued up. It's changed the structure of how my projects work a bit, and do you have to do some homework fast of how slow/fast your various providers or local inference are, but it's worth it. Start off with hobby projects so you get a feel for how it works.

I left something gargantuan running over the weekend (decompiling 1980s-era system software) and look forward to checking it out later today when I have a few free minutes.

[zozbot234]

In principle, a slower inference ought to be easier to steer and course-correct. You'd always be able to look at partial results, especially with a local model that doesn't hide its thinking.

[schmorptron]

it has to be so bursty for realtime usecases like chat, which is what most people are using it for today. of course, once (if) stuff like software dark factories start working out for the average person, then you'll be able to make full use of your hardware for workload where asynchonous execution is feasible and have it run several parallel tasks overnight, with an orchestrator managing the gpu(s) allocations.

[zozbot234]

Chat doesn't have to literally be realtime though, that's just the model most users have settled on. You could fire off your request, let it work unattended and check back on it later (perhaps after getting some notification from the chat frontend via RSS, Web Notifications API or similar that the full response is ready).

[DennisP]

For software, not only that, but run batches with a cluster of agents working different parts of the same task. Software like Yegge's Gas Town has one agent act as "mayor," and others work on writing or testing various pieces, with all the agents messaging each other. In his book Yegge writes about using up to thirty agents at once.

[usef-]

If your workload fits long batches throughout the night you could schedule them better, yeah. But I think very few have a usage pattern like that?

Given the hardware shortage in the world, I suspect renting ("sharing") via APIs will likely remain cheaper for the foreseeable future since each piece of hardware isn't sitting idle nearly as much.

[soerxpso]

A decentralized inference network would be cool. Something that's set up so that I can run a model for personal use on beefy hardware, but also farm out the unused GPU time to the network, probably at much lower prices than normal providers since it would be slower and would lack data security guarantees.

[jacobpfeifer]

Not quite what you are asking for. Instead of selling the unused GPU, you're donating it to the common good.

https://cocore.dev/