The author was able to use the MJPEG output of the USB HDMI capture devices to get low latency video.
https://github.com/pikvm/ustreamer