I had that view of physics initially, despite being very interested in computers and mathematics, because I viewed physics as a really inelegant memorization-based field, that consisted of filing away in your head a bunch of special-case equations for things like "how to determine final position with constant acceleration and constant friction". I disliked it much less when I took a calculus-based physics course in college, where all those miscellaneous equations were exposed as special cases of a few more general relationships.
So all the people who are incapable of calculus can't appreciate pysics? That makes sense. It also restricts appreciation of physics to the top quartile, conservatively.
I don't really know in general; I personally didn't appreciate physics when it was presented to me algebraically in high school, despite getting an A in it. Perhaps it'd be possible with a different way of teaching it; not sure.
Programming is essentially not taught in U.S. high schools (it's sometimes available as an elective which most people don't take and it's almost never taught by someone who had any background in software development).
When I tell most people that I'm a computer programmer, I get blank stares because they just don't have any concept of what it entails. It's completely magic to them and I usually end up describing it as analyzing business problems and translating those requirements into software, but they still don't have a clue as to the actual mechanics of programming since they've never been exposed to it in real life. That, or they ask me if I could take a look at their router and then I have to explain "that's not what I do".
It's probably a "grass is always greener" situation, but I'd often enjoy being able to relate my career to average people who share a minimal amount of common understanding, even if it's just to tell me they hate it. I don't expect education to make people like a subject.