[schaefer]

Amazing writeup. And what a heavy hitter of an inaugural blog entry...

This might be the right time to ask: So, on the one hand, this is what it takes to pack 192gb of Nvidia flavored vram into a home server.

I'm curious, is there any hope of doing any interesting work on a MacBook Pro Which currently can be max-spaced at 128 GB of unified memory (for the low, low price of $4.7k).

I know there's no hope of running cuda on the macbook, and I'm clearly out of my depth here. But the possibly naive day-dream of tossing a massive LLM into a backpack is alluring...

[Eisenstein]

Download kobodlcpp and give it a try. It is a single exec and uses metal acceleration with an Apple Arm CPU.

[schaefer]

Thanks!