[bitwize]

It may be taught that way, but do kids learn it that way? Kids learn a bunch of "facts", and are tested on their capability to repeat those facts in a timely manner. Then they are taught rote-level procedures for combining those facts in order to do arithmetic on multi-digit numbers. The fact that subtraction is the opposite of addition gets demoted to "interesting bit of trivia".

For most school kids, algebra is the first place where the art of mathematics comes into play: where you are given a problem, and are not told specifically how to get from here to a solution. You have to figure that out yourself with the aid of mathematical tools.

For most people, who aren't used to the art of logical thinking, that is crushing.

[saraid216]

Slightly tangential, but every time I work with someone on basic kinematics and they say, "Oh, well, if I just flip the sign from positive to negative, I get the right answer! I'm done!" a little piece of me dies.

It's like... you're not even trying to understand the concepts, are you?

[matho]

I think this is hard because it's not immediately obvious that direction is such an important concept in kinematics. It's probably the most important thing to get across when teaching the subject, but you have to battle against prior knowledge:

Math students (that is, everyone) earlier learned that this sign-flipping strategy is ok and perhaps an encouraged shortcut when e.g. subtracting.

(see also that Khan Academy video - "first do the multiplication, then think about the sign")