Do you have just 1 varnish instance or two?
I never did fully solve the ban propagation problem when running multiple redundant varnish instances.