I think your opinion is more and more popular. Personally I drive for a hobby, I don't have a commute or anything. Is my few thousand miles of year hobby driving really the issue?
There's no single thing you can point to as "the issue" (unless you just point to fossil fuels in general, I guess). Driving a couple of thousand miles per year as a hobby is pretty bad, that's quite a lot of "unnecessary" co2 emissions. Will it cause mass starvation and make environments uninhabitable? No, not by itself, but it'll contribute more than most hobbies. I believe it's in the same ballpark as a round-trip flight across the Atlantic, for reference.