You can .gitignore everything. I used to have my homedir as git repo, but abadoned that because it interfered with other git repos inside my home. Nowadays i have ~/.config as git root, with some symlinks supporting the programs that don't look there on their own.