Dead of winter, if the waves are pumping, it will be packed all day (surprising how everyone manages to be unemployed when the waves come).
January/February the vast majority are in a 4/3 wetsuit with booties (to protect the feets), but really it's not _that_ cold, nothing compared to northern Europe (where the poor bastards rub vaseline on their faces) or northern Cali in the winter, which is not only colder, but has a much larger issue, the man in the gray suit.
Don't stay there in the winter though, the lack of people and activity can be a bit depressing.