I think I used it once: I wanted to talk to a man on a swamp and I needed one charm to talk to him.
Be careful about them, since they seem to be finite in supply. Though I don't losing all of them and not being able to talk to that man would make any discernable difference in the long run.
Yeah, as far as I could tell, you only used them to talk to the old man in the swamp - you'll find them periodically in your traps, though - I had 8 or 10 by the time I finished the game, so don't worry too much about losing them.
Be careful about them, since they seem to be finite in supply. Though I don't losing all of them and not being able to talk to that man would make any discernable difference in the long run.