You should check out Git LFS if you want to do that, as it sounds like a good idea in the first place!