You don't need to drop Facebook -- just turn privacy settings so that your face can't be identified publicly or beyond 1 degree of connection. People should know how to do at least that.
Your main profile picture is always public, as is your cover photo, so you will have to use a picture that doesn't identify you, which may be confusing to your friends.