You probably should revoke all your existing credentials and replace them with fresh ones as soon as you pull them out of the VCS. That way, your attackers have the credentials, but they don't work anymore.
We've left test keys in our git repos. They don't work for anything except a virtual machine used for local development, but I always thought it would be amusing if a hacker grabbed them and got frustrated trying to use them.
So you leave them in as a kind of poisoned honeypot. They'll go right to the wrong info....
I'm sure you've daydreamed of the facial expression of the scriptkiddie the moment he stumbles across your fake keys illicitly, only to be disappointed hours of unfruitful hacking later :)