Arguably what Urban Dictionary is doing is to weigh by "net favorability" in some sense and quantity of votes. Quantity of votes correlates to relevance, particularly because UD is meant to represent popular usage.
We actually switched to Wilson score. Doing it later has some weird effects, when you've already have a lot of people typically voting on the first definition, and then suddenly the order gets switched because something has a higher ratio giving it higher confidence. We're honestly not sure it's done anything that great for UD, sometimes simple is just better.
You might want to weigh less controversial (as in abs(upvotes - downvotes) higher. This would be somewhat like Effect Size in science.
The Bayesian approach would be to assume the true vote distribution is binomial and use a beta prior (possibly with Jeffrey's degenerate bimodal prior). Then as the total number of votes increases the posterior distribution tightens. Ranking score is prob(score>0).
Would a better idea for UD be like the original Facebook "like" system? So you only vote if you think it's relevant and only popular definitions sit at the top.