I don't think it's that at all. Pun threads generally make for very boring reading/discussion and tend to rise to the top because they're quick to digest, uncontroversial and mildly amusing. If you're looking for more substantial comments (arguably most HN users are) these can be a real chore to slog through.
You misspelt "people who like intelligent discussion". A joke now and then is okay, but try to give it enough redeeming value. (And for the rest, don't worry too much about downvotes.)