I think it used to just mean "singular", from the Latin grex, gregis meaning herd, and e/ex meaning "out of". It could mean singularly bad or singularly good I guess in English, but in Latin I think it had more of a connotation of exceptional, extraordinary, eminent.
I tend to assume that anyone who objects to “I could care less” has never lived in the New York City area. See the mention of Yiddish in the above link. But for some who object to it, that’s the issue: it’s a shibboleth of a culture they’re not part of.