Hacker News new | ask | show | jobs
by najarvg 9 days ago
Do diffusion models support tool calls? If so is the tool call support on par with autoregressive models or worse? (edited spelling)
2 comments

Any text generation model can easily be made to support tool calls.
omlx.server - WARNING - POST /v1/chat/completions -> 400: Tool calling is not supported with diffusion models.
Pull request #1837 that enables tool calls on supported diffusion models was merged as 7c1971e today. I previously tested mlx-community/diffusiongemma-26B-A4B-it-8bit on a custom patched version of omlx in the Zed Agent Panel. The majority of the tool calls worked.

What didn't work reliably was specifically write tool calls and this is not resolved by the pull request. But as far as I understand the problem is not the inference framework but the root issue is that DiffusionGemma emits incorrect JSON.

When `content` contains `, ` inside a string value, the decoder splits there and emits the remainder as a nonsensical JSON key. So `{"path": "f.py", "content": "def f(x, y):\n return x"}` becomes `{"path": "f.py", "content": "def f(x", "y):\n return x": ...}`.

I wondered if the JSON issue might be related to quantization and tested the BF16 variant of google/diffusiongemma-26b-a4b-it via NVIDIA NIM. The model did not show the delimiter-splitting bug. It did however have a quote-handling issue. Among others it duplicated tripple quotes (`"""..."""` becomes `""""""...""""""`).