The condition is prosopagnosia: https://en.wikipedia.org/wiki/Prosopagnosia and the wiki page says Saks himself suffered from it. It is not that rare, though many people that have it just think they are "bad with faces" or have bad memory when recognizing faces (even though otherwise their memory may be excellent). Many people that have it learn to "route around" it by explicitly memorizing facial cues (hairstyle, facial features, hair color) or other cues (clothing style, voice, body characteristics, etc.) and going off context ("if I meet this person in the office, this is one of my coworkers"). If the person is encountered out of context and in unusual attire/situation, they may not have enough clues to trigger recognition - so if somebody you know doesn't recognize you on the street, maybe they are not necessarily rude or avoiding you but maybe just have some prosopagnosia. :)
Yup, one of my professors in college had this and she would use a different variation of her name in each facet of her life so that when she met someone out of context, she would know where she knew them from.