You can also just place config files anywhere if you know what you then load. That's what I do in my dotfiles, but not exactly like the parent. I also purposefully keep the repo size tiny so it's also just easy to clone. I'd recommend setting a env var so you can always just set that
Also don't forget you can have local vim files. I have a function at the end of my vimrc that looks for '.exrc', '.vim.local', '.nvim.local' in the current directory. Helpful for setting project settings.
I've found lnk [0] to be a nice tool for this. Similar to GNU Stow as another comment mentioned, but plays a bit nicer with git (and, in my opinion, is nicer to use).
Edit: just remembered there was a good comparison of lnk and stow on the HN discussion of lnk from a few months back [1].
It's kinda amusing how much of interesting software there is beyond coreutils and GCC that came from GNU, and how little adoption it has actually seen.