That's nice, until you accidentally type "git clean -fdx" into the wrong terminal and nuke your home directory. I would be way too scared to have all my files be untracked in an unclean repo.
I use `git clean` maybe once per year on average, so very low risk for me. And if that ever happens... that's why I have hourly backup snapshots! If I didn't trust my backups, I'd definitely address that before worrying about dotfile management. If you don't have a good safety net then you end up reinventing the net 100 times in other ways and everything gets more complicated.