Well, a smartphone is fairly bad for its small size, but we're talking less than 100kg of CO2[1][2] to make and operate for its lifetime. But driving a car for a year in the US produces 4600kg[3]...
My understanding is that CO2 output is the least worrying part of smartphone usage. It's all the other toxic waste released during mining. Also, if cars are more polluting, that just means cars are bad too, not that smartphones are magically ok.
You are right. Few do. If people cared about the environment that much, they would have lobbied for laws against planned obsolescence. Me buying 5 phones or even 500 won't make a difference to the environment. Laws forcing phone makers to support their phones for a minimum of x years might. Clearly, our society is not interested in that. Why should I then go out of my way to inconvenience myself to do something that will make zero difference? No reason to at all.
Typically price is a pretty good indicator of the amount of resources that went into making something. I'd expect the environmental cost of an iphone to be a similar multiple of the environmental cost of an android.
1: https://www.cnet.com/news/apple-iphone-x-environmental-repor...
2: https://sustainability.google/reports
3: https://www.epa.gov/greenvehicles/greenhouse-gas-emissions-t...