[d4rkp4ttern]

I have an M1 MBP 64GB. Can I run it on my M1 Or do I need a GPU ?

[swsdsailor]

I got it to work with MPS by having pytorch with mps support and then editing the cli.py file to allow the use of mps:

Allow passing in --device="mps": ie: choices=["cuda", "cpu", "mps"]

Set kwargs: kwargs = { "torch_dtype": torch.float16 }

then adding to("mps") on line 98: model = AutoModelForCausalLM.from_pretrained(model_name, low_cpu_mem_usage=True, *kwargs).to('mps')

commenting out: raise ValueError(f"Invalid device: {args.device}")

and changing cuda to mps on line 80: if args.device == "mps":

I'm not sure it's working correctly but at least it's a step. It's told me how to catch a duck but it often falls into some "renewable energy" sequence. :D

[pugio]

I have it running, slowly, on the same machine. I would love for someone to get support running for MPS backend (The GPU) but it does run on the CPU.

[ThorsBane]

Thanks for sharing. How slowly is slowly? Do you anticipate that an M2 Max with 96gb of memory would run it noticeably faster?