Because clicking a mouse takes you away from the keyboard .. a context switch if you like, which added up over time slow you down when you have work to do.
Perhaps, but it'd be less of a context switch than remembering what directory I'm in and where it was that I wanted to go. It may be that I'm growing old and dull, but I'm better at clicking on things than remembering them.