That's strange since the rubber on my Nexus 4 charger is very tacky, almost too sticky. If your phone is slipping then try cleaning the dust off the rubber on the charger to make it tacky again.
How long have you had it? The rubber on mine was very sticky at first, but now it just slips off. (I've had it since November) I'm wondering if there is some type of adhesive that comes off.
I've had mine since May. Probably the rubber coating can be destroyed if you clean it too aggressively. Don't rub it with a cloth or an anti-static wipe. That will just cause small fibers to stick to the tacky rubber.