After you are done swimming, you are likely to remove the goggles before you completely dried yourself. Water can run from your hair down to your eyes.
Also, after swimming for an hour and taking a break, I like to remove them for a bit. It has happened that I got water in my eye in this way. Never had an infection because of it though!