NM doesn't connect to wifi by itself, it uses wpa supplicant for that. If something is slow, it is wpa supplicant. Fortunately, nm backends are modular and you can use Intel's iwd instead.
A while ago, maybe about a year ago, after upgrading Debian I discovered that I was no longer able to connect to my 5GHz wifi network, but could connect to 2.4GHz networks. Switching from NM to wicd fixed the problem. Same kernel, same wifi card and driver, same wifi AP with the same configuration. As baffling as it seems, getting rid of NM was the only thing I had to do, or could do, to make it work again. NM is now dead to me.