[pantaloony]

Exactly. A lot of what's available to men these days is former athletic/sporting wear turned "athleisure", then sometimes losing much of the original association: button-down collar shirts (there's a reason they're traditionally made of relatively tough Oxford cloth, and the collar's buttoned down so it stays that way during vigorous activity), polos (invented for tennis), sport coats (hunting, riding), "rugby" shirts (obviously).

Then our most popular outerwear shirt, the T-shirt, used to be underwear with a strong military association, and came into common use the same way much of the rest of our modern fashion did: youth culture, which AFAIK is also how we got the blazer (dad's orphaned suit jackets) and two-button and especially two-roll-three suit jacket styles (kids wearing hand-me-down 3 and 4 button suits and orphaned jackets "casually" with fewer buttons done up).

[DanBC]

> and the collar's buttoned down so it stays that way during vigorous activity)

Yes, but also the collar is buttoned down because those shirts don't have bones and are not starched.