Men do have trouble hearing women

Men who are accused of never listening by women now have an excuse -
women's voices are more difficult for men to listen to than other men's.
Reports say researchers at Sheffield University in northern England have
discovered startling differences in the way the brain responds to male and
female sounds.

The research shows men decipher female voices using the auditory part of
the brain that processes music, while male voices engage a simpler mechanism.

The Daily Mail quotes researcher Michael Hunter as saying, "the female
voice is actually more complex than the male voice, due to differences in
the size and shape of the vocal cords and larynx between men and women, and
also due to women having greater natural 'melody' in their voices".
"This causes a more complex range of sound frequencies than in a male
voice," Mr Hunter said. The report says the findings may help explain why
people suffering hallucinations usually hear male voices - the brain may
find it much harder to conjure up a false female voice accurately than a
false male voice. The research is published in the specialist magazine
NeuroImage.