Yeah, this is one of the reasons I like HN. Memes are fun, but sometimes I just want to discuss stuff seriously. I'll still upvote a clever pun or subtle meme reference on HN, but no reason it should be at the top.
I love puns, but if a comment is only made for the sake of making the pun, it probably isn't adding anything to the discussion. A pun or meme-ish comment with additional actual content however, is much more conducive to conversation, and doesn't risk derailing or dead-ending an otherwise perfectly good thread.