[crazymoka]

Why do you need mlx? Like your blog post by you never explain why things need to be used.

Why isn't using localforge enough as it ties into models?

[avetiszakharyan]

I was just trying to make sure is maximally performant, and did it with MLX because i am running on mac hardware and wanted to be able to run 30b in reasonable time so it can actually autonomously code something. Otherwise there are many ways to do it!

[p0w3n3d]

It would be nice if you mentioned it's about apple silicon, and not apple intel computers. They're still ubiquitous nowadays

[Tokumei-no-hito]

we're on the 4th generation of silicon now

[turnsout]

I believe mlx will allow you to run the models marginally faster (per a recent blog post by @simonw)

[simonw]

Yeah, you don't necessarily need it but it's optimized for Apple Silicon and in my experience feels like it gives slightly better performance than GGUFs. I really need to formally measure that so I'm not just running on vibes!

[indigodaddy]

I for one, am willing to just trust you bro ;)

[turnsout]

Yeah I’ll go with Simon’s vibes over most people’s measurements!

[freeone3000]

mlx is an alternative model format to GGUF. It executes natively on apple silicon using Apple's AI accelerator, rather than through GGUF as a compute shader(!). It's faster and uses fewer resources on Apple devices.