If by muscular and skinny you mean that one has more capacity for developping strength, then sure, why not ? Should our expectations not be based on ability ?
No, I meant exactly what you wrote. You say that since physical strength of men and women differ their cognitive abilities should differ too. The physical strength of muscular and skinny men differ too, hence, according to your argument, their cognitive abilities should differ too.