[colossal]

You need to switch the drivers from WDM to ASIO. Most DAWs have this as an explicit choice with WDM as the default.

[adzm]

I think WASAPI has comparable latency to ASIO drivers. I can get around 20-30ms latency with WASAPI though ASIO takes me down to 10-15. Granted I'm using a dedicated USB audio interface though.

[namibj]

You should be able to easily sustain below 5ms buffer size, and RT_PREEMPT Linux has no problems keeping a few hundred microseconds of buffer filled (a suitable PCIe soundcard should easily make 100us buffer level reliable; that's 9.6 samples at 96kHz sample rate).

Yes, sub-ms latency is hard. But the infrastructure exists.