Yeah, I've had similar experiences to yours, nordic ski racing on a team all throughout college. We'd all get coughs the first few cold days, but that was it. We would avoid doing interval workouts if it was below 0° F though.
Yeah, when it was really cold (<= 0F) were were required to wear facemasks (those neoprene masks with the nose fold, paper-punch holes at the mouth, and velcro in the back[1]) to prevent frostbite on the face, and they definitely helped with breathing as well.
[1] https://www.amazon.com/Tough-Headwear-Neoprene-Ski-Mask/dp/B...