Because most people care very much about social relationships, and like it or not, other people just naturally tend to be put off by things that look strange or unattractive. You might decide you genuinely don't care what anyone thinks of you, but realistically speaking that's a somewhat unusual stance.
I get that, but the way people talked about it, I though it was something really horrible. And it turns out it is just a little different. I have used some way more strange stuff in this life of mine.
That's one of the more important factors you have to consider when creating a device that you want to market to the general public. Tons of people, don't wear bluetooth headsets because of how bad they look. Sure there are some headsets that look decent, but the headsets already have a certain label associated with them. Wearing them is not cool. Google has to really do this right from the start or else Glass will just be another gadget that only geeks wear.