Could you point to some examples where this has worked? It seems that the entry fee would either 1. limit the community's growth severely (not necessarily a bad thing), or 2. filter out good candidates.
SomethingAwful's forums required a paid membership and have been around a long time. IMO one of the points of a paid model is keeping membership down to people who actually care enough to pay said membership.
Additionally, the implicit threat of having your paid membership revoked for bad behavior can provide a natural check.
Something Awful has used this to great effect to keep trolls in line. I wonder what % of their membership revenue comes from banned users purchasing new accounts.
I'd say most of it, at this point. As a long-time goon, getting banned and rereging the account is a way of patronage. No joke, it's very common behaviour there.