The page should probably hide the score of the color palette you're currently rating. After you notice it, you can easily be influenced by it. Showing it after voting would be interesting though.