[tlb]

I looked at a few systems but didn't think any of them would work.

The real-time requirement is fairly hard. The robots I work on use a 1 kHz feedback loop, so it has 1 mS in which to recalculate everything.

Caffe is pretty efficient if you can supply batches of values, but the overhead is high when running real-time.

[EquallyJust]

have you tried torchscript w/ the C++ interface?

[tlb]

No. It looks promising. I should try it.

[sangfroid_bio]

I think Tensorflow with Swift or Rust bindings may also work.