I wouldn't say so, if they're properly balanced / supported. Most headphones are heavier than that and comfortable. Ski goggles are around the same weight, and you can easily wear them for long periods without even noticing.
It makes a difference. People are less willing to wear heavy headphones all day than they are willing to wear regular glasses all day. They also wouldn’t want to wear ski goggles all day (unless they are into all-day skiing I guess). I much prefer wearing 20-gram glasses to wearing 35-gram glasses. 100-gram glasses would be a turn-off.
The difference is that glasses sit on your nose and headphones sit on your head. 100g headphones are considered lightweight. Apple's headphones are 386 grams, which are too heavy for a lot of people.