+1. In Ableton on Windows you can get your latency down to ~40ms without a dedicated sound card using ASIO. Mac's drivers are even better with sub ~20 ms on my m2 pro IIRC.
+1 to the comments here. Part of the issue here is running these applications in Python. It's not really optimized to handle these loads and do DSP-based compute efficiently.
You seem surprised, but any sort of live production requires this. Check out SonoBus, it achieves adequately low end-to-end latency even with network delays in the mix.