[quietbritishjim]

This is south London.

But also, it's just obvious from the words in the phrase. I assume that outside toilets were often just wooden shacks, so a brick one is notable for its robustness. Is that comparible with a beefy labourer's physique? Sure. Is comparible to a curvy, classically feminine body shape? That doesn't make sense.

As the sibling comment to this one says, it doesn't even just apply to people (in fact I'd say it primarily doesn't).