[Jenda_]

> Over USB, the control message latency leads to this taking tens of milliseconds.

This sounds an order of magnitude wrong. I have just setup a loopback with a CH340 USB to serial adapter and ran the following code:

  #!/usr/bin/python3
  import serial
  import time
  
  ser = serial.Serial(port="/dev/ttyUSB0", baudrate=1_000_000, timeout=1)
  
  iters = 100
  x = time.time()
  for i in range(iters):
    towrite = b"%i\n"%i
    ser.write(towrite)
    line = ser.readline()
    assert(line == towrite)
  delta_ms = (time.time() - x)*1000

  print("Finished %i iterations in %ims = %.1fms/iteration"%(iters, delta_ms, delta_ms/iters))

and it says Finished 100 iterations in 272ms = 2.7ms/iteration