Well, yes, lady bits are also used pejoratively! But the male one is acceptable in (most) casual conversation, way less taboo.