[eyeris]

Agreed. Same with intel’s NPUs. I’ve been testing with my intel core evo 155x. The npu only runs int8 as well. At least, Intel has put in a decent amount of effort into the ecosystem

There are a couple ways to interface — DirectML by MS and Intel’s native api (they provide OpenVINO model conversion to convert normal Python ml models) I’ve tried ONNXRuntime conversions for both backends to little success. Additionally the OpenVINO model conversion seems to break the model if the model small enough.

OpenVINO model server seems pretty polished and has openapi compatible endpoints.