[jonathaneunice]

Are the Apple neural engines even a practical target of LLMs?

Maybe not strictly impossible, but ANE was designed with an earlier, pre-LLM style of ML. Running LLMs on ANE (e.g. via Core ML) possible in theory, but the substantial model conversion and custom hardware tuning required makes for a high hurdle IRL. The LLM ecosystem standardized around CPU/GPU execution, and to date at least seems unwilling to devote resources to ANE. Even Apple's MLX framework has no ANE support. There are models ANE runs well, but LLMs do not seem to be among them.

[Yukonv]

It is possible but requires a very specific model design to utilize. As this reverse engineering effort has shown [0] "The ANE is not a GPU. It’s not a CPU. It’s a graph execution engine." To build one requires using a specific pipeline specifically for CoreML [1].

[0] https://maderix.substack.com/p/inside-the-m4-apple-neural-en... [1] https://developer.apple.com/documentation/coreml

[jonathaneunice]

That's the best "what is ANE, really?" investigation / explanation I've seen. Directly lays out why LLMs aren't an ideal fit, its "convolution engine" architecture, the need for feeding ANE deep operation sequence plans / graphs (and the right data sizes) to get full performance, the fanciful nature of Apple's performance claims (~2x actually achievable, natch), and the (superior!) hard power gating... just _oodles_ of insight.

[krackers]

More info on specific design choices needed to run models here [1]. I mean it is possible given that apple themselves did it in [2], but it's also not as general purpose or flexible as a GPU.

[1] https://news.ycombinator.com/item?id=43881692 [2] https://machinelearning.apple.com/research/neural-engine-tra...

[dust42]

There is a project on github named ANEMLL. Was discussed here a month ago, running LLMs on iPhone - https://news.ycombinator.com/item?id=47490070