To follow up on what mifchip said, binary data can possibly be quite large, and git-annex was designed to manage files in git and create syncing mechanisms without the disk space and overhead of managing the file contents inside said git repo. I would recommend that over using straight git, as it also has options for encryption which might simplify the workflow or make things more transparent. Again, ask around for better info regarding how versioning the file can be used by an adversary (it does depend on the encryption algorithm and implementation, YMMV), but this might be a more suitable solution for you.
http://git-annex.branchable.com/
http://git-annex.branchable.com/encryption/