The problem I have with nvm is speed (running it at shell startup introduces a very noticeable delay), and integration with other shells - it's always been a pain to get it running with fish.
I generally use only a single node version, and therefore don't use the nvm command much, so I've solved the slow shell init by not loading nvm at all, and instead, just set the PATH.
idk about zsh, but for fish there is https://github.com/FabioAntunes/fish-nvm which delays executing nvm until you use one of the node binaries. Essentially, node is aliased to `nvm activate && node`