How long have you had it? The rubber on mine was very sticky at first, but now it just slips off. (I've had it since November) I'm wondering if there is some type of adhesive that comes off.
I've had mine since May. Probably the rubber coating can be destroyed if you clean it too aggressively. Don't rub it with a cloth or an anti-static wipe. That will just cause small fibers to stick to the tacky rubber.
How to clean it: http://itecsoftware.com/how-to-clean-your-nexus-wireless-cha...
In my case the charger sometimes sticks to the phone when I pick it up.