OK, I should probably amend that. Verbal fillers and overused words are often more noticeable to the audience than to the speaker (or umm-ahh-er). Those tend to not be judged very harshly though and are rarely a reason to be fearful of how you're perceived while public speaking.