Teaching young kids is a genuinely hard job which is severely underpaid in almost all countries. I think that is the primary reason for men not flocking into teaching.
That reason isn't limited to men. It's why people in general don't get into teaching. Many more men and women would be interested in teaching if it paid better, and we'd have better teachers.