Some items are "condition rarities" -- it's worth a fortune in never used condition, but nothing special when worn.
Other items are rare as long as they can be reliably attributed. You could run a 1804 US dollar over with a locomotive and it would still be worth six or seven figures as long as the date could be identified. I'd expect plenty of the "rare" stamps would hold a significant slice of value even used. Now, the fact you gave up possession of the expensive item is another issue.
Returning to main HN stamping grounds: that's what makes some of these "new collectible" schemes sort of weird (i. e. professionally graded/certified video games)-- they're not necessarily trading on inherently rare items, just creating a "condition scarcity" economy that previously didn't really exist. It's not "the only one of its kind", it's "the only one the grading service gave 9.452345/10".