I did not realize this until I cancelled a card that I'd had for 15-20 years, much longer than any others, because I'd switched to another card from the same company (AMEX) and wasn't using that old card anymore. My score went down quite a bit, and I was so upset at Amex for not telling me when I was trying to cancel that old card. I would have just kept it and not used it.
If you close a particularly young card, it can also bring your score up if the others are older. Last I looked it used an average for the age of all your credit lines.