[billylo]

Windows and macOS does come with a small model for generating text completion. You can write a wrapper for your own TUI to access them platform agnostically.

For consistent LLM behaviour, you can use ollama api with your model of choice to generate. https://docs.ollama.com/api/generate

Chrome has a built-in Gemini Nano too. But there isn't an official way to use it outside chrome yet.

[vintagedave]

Do you know what it’s called, at least on Windows? I’m struggling to find API docs.

When I asked AI it said no such inbuilt model exists (possibly a knowledge date cutoff issue.)

[billylo]

https://learn.microsoft.com/en-us/windows/ai/apis/phi-silica

[vintagedave]

Thankyou!

[bredren]

Yes. I am not aware of a model shipping with Windows nor announced plans to do so. Microsoft’s been focused on cloud based LLM services.

[usefulposter]

This thread is full of hallucinations ;)

[tony_cannistra]

These are the on-device model APIs for apple: https://developer.apple.com/documentation/foundationmodels

[nvader]

Is there a Linux-y standard brewing?

[billylo]

Each distro is doing their own thing. If you are targeting Linux mainly, I would suggest to code it on top of ollama or LiteLLM

[1bpp]

Windows doesn't?