That's strange. They automatically close for me and I didn't change any settings anywhere.
I would be nice if they were just Growl notifications however.