So there are two ways I can see to fix this. Either make setting the time always requires a password, or, add a signal that time-sensitive processes can listen to that gets tripped whenever time is altered.
There's a much simpler fix that is local to sudo. Sudo has to make the decision of whether to require a password. Just change the line that says something like:
if (current_time - last_password_time > INTERVAL) require_password();