[abhorrence]

You can do it the other way around. If the signature seems fishy, you set it aside unopened, and only do further validation before opening it if the election is close enough for it to matter.

For example, in an election with 100 votes cast. If 10 signatures are suspicious, you only do extended validation and open them if the election was won by a margin smaller than 10.