[radlad]

It is much slower than nVidia, but for a lot of personal-use LLM scenarios, it's very workable. And it doesn't need to be anywhere near as fast considering it's really the only viable (affordable) option for private, local inference, besides building a server like this, which is no faster: https://news.ycombinator.com/item?id=42897205

[bastardoperator]

It's fast enough for me to cancel monthly AI services on a mac mini m4 max.

[diggan]

Could you maybe share a lightweight benchmark where you share the exact model (+ quantization if you're using that) + runtime + used settings and how much tokens/second you're getting? Or just like a log of the entire run with the stats, if you're using something like llama.cpp, LMDesktop or ollama?

Also, would be neat if you could say what AI services you were subscribed to, there is a huge difference between paid Claude subscription and the OpenAI Pro subscription for example, both in terms of cost and the quality of responses.

[lostmsu]

Hm, the AI services over 5 years cost half of m4 max minimal configuration which can barely run severely lobotomized LLaMA 70B. And they provide significantly better models.

[Matl]

Sure, with something like Kagi you even get many models to choose from for a relatively low price, but not everybody likes to send over their codebase and documents to OpenAI.

[nomel]

It's probably much worse than that, with the falling prices of compute.

[staticman2]

Smaller, dumber models are faster than bigger, slower ones.

What model do you find fast enough and smart enough?

[Matl]

Not OP but I am finding the Qwen 2.5 32b distilled with DeepSeek R1 model to be a good speed/smartness ratio on the M4 Pro Mac Mini.

[bastardoperator]

I'm running the same exact models.

[a1o]

How much RAM?

[Matl]

It takes between 22GB-37GB depending on the context size etc. from what I've observed.

[a1o]

Thanks!

[jamesy0ung]

I presume you're using the Pro, not the Max.

Anyways, what ram config, and what model are you using?

[fetus8]

How much RAM are you running on?