When people say they can’t draw, they aren’t referring to being able to draw a simple geometric shape like a letter. They mean a portrait or a still life or landscape or something.
It's the same skill, just a different degree of practice. Depending on your skill level drawing involves a lot of simple geometric shapes and then using those shapes as a point of reference to draw the final result. The reason why art seems so difficult is because people don't see the intermediate steps which were taken, they only see the final result.
That's not what I'm saying. I understand they are saying they can't draw a portrait but what I'm saying is the only reason is because the skills usually weren't taught past a certain level. Those same geometric shapes are used in the construction of everything else.