They do make sense for many users, and they are the closest you can find to a proper graphical representation of languages. When I add a language that I know to be official in several countries, I look at my analytics to see where most users come from and use the flag from their country. I can't remember a time where it did not also match the country with the most speakers.
It's a common enough practice that most people usually know what it means, but there's a reason you don't see flags on Wikipedia, Facebook, or Youtube. Languages are spoken in many countries, and countries are multilingual. There are quite a few articles around the web on this topic, but that's basically what they boil down to: languages are not countries. Some users may be confused or offended that their flag is not represented.
And as a Canadian I find it generally a little weird that the Canadian flag often means Canadian French, and I have to click the US flag to get English (which is of course a slightly different English than Canadian English which is probably unavailable).
I guess it's something like "language most unique to that country", no but that's not right either... I don't know.
Unless you have different pricing per country or something orthogonal to language, I'm sure than a speaker of Canadian French can figure out that clicking the French flag may help them understand this page better. It's a common enough idiom on the web.
I think in the case of more than one country per language, you're right, just picking a big and/or well-known country as "representative" is fine: French flag for French, US or UK flag for English, German flag for German.
The bigger problem is the other situation, of more than one language per country. India has ~13 languages with >10m native speakers, and using the Indian flag for all of them would be pretty confusing. You could pick state flags (e.g. the flag of Gujarat to represent Gujarati), but that can be a politically tricky issue. In some cases choosing a representative flag for a language has even stronger political overtones, like using the flag of the Kurdistan independence movement to represent the Kurdish language. Plus, it's not always that clear which flag to pick, and user recognition may not be as high as in the French-flag-for-French case.