I was tempted to the the same thing as OP, but decided to go with stow, because its concept is more sound than whatever I may concoct. Currently, my dotfiles are packages (folder) which targets $HOME. I clone the repo to $HOME/dotfiles, then I can use `stow <package>` inside the folder to install each package easily. I treat it like dpkg|rpm|... but for my configuration. I'm trying to make each package its own thing, like mail, sway, i3, etc,...
I don't have any experience with chezmoi. I've managed my dotfiles manually and with the bare git repo trick, but I've looked at chezmoi's docs and it looks like ansible, but for your $HOME. I only have a few computers I want my dotfiles on and stow has been perfect for having my dotfiles in a single folder and linking them where I want to be. Only a few of these configs I'd want somewhere else and I'd either create a new repo (work computers) or copy the files (random place)