[jjuliano]

Currently, Docker does not support Metal GPUs.

When running LLMs on Docker with an Apple M3 or M4 chip, they will operate in CPU mode regardless of the chip's class, as Docker only supports Nvidia and Radeon GPUs.

If you're developing LLMs on Docker, consider getting a Framework laptop with an Nvidia or Radeon GPU instead.

Source: I develop an AI agent framework that runs LLMs inside Docker on an M3 Max (https://kdeps.com).

[rtwld]

Podman does support GLU acceleration through libkrun with virtio-gpu (venus) on Mac: https://podman-desktop.io/docs/podman/gpu

[asadalt]

how is it in practice (if you have tried). I have some vulkan work and I am too lazy to setup a new ec2 for it.