A new paper in The American Journal of Humans Genetics, The Divergence of Neandertal and Modern Human Y Chromosomes, reports on possible reasons why we don’t see Y chromosomes in modern humans from this archaic lineage, despite exhibiting detectable levels of autosomal admixture. As you might recall the clear lack of deep branching Y and mtDNA lineages was long one of the major genetic rationales for why gene flow between Neanderthals and modern humans was presumably not very significant. This, despite suggestive evidence from morphological analysis as well as inferences from autosomal data. The problem is that it is harder to do the sort of clean phylogenetic reconstruction via a coalescent model utilizing autosomal data (which recombines, as opposed to the Y and mtDNA, which do not for the regions of interest), so ancient genome sequences were really what was needed to convince most people with these sorts of markers.
This makes us ask: why are Neanderthal Y and mtDNA lineages not found in modern humans which exhibit indications of gene flow from other hominin lineages? After all, the lack of these really led many people off on the wrong track for years. I recall in 2008 going to a talk by Svante Paabo who reported that the Neanderthal mtDNA he had sequenced was definitely very different from anything in the current databases for our species, which confirmed his assumption that there was no admixture into modern populations (Paabo changed his tune very soon after due to the whole genome sequencing obviously). One simple explanation is that because effective population sizes of Y and mtDNA are smaller than autosomal regions of the genome they’ll be more strongly subject to drift, and exhibit higher extinction rates. In other words, it wouldn’t be that surprising of all Neanderthal Y and mtDNA went extinct after admixture because they were a small minority, and most lineages went extinct in any case. Researchers who work in non-human phylogeography who relied on mtDNA in particular can tell of many stories of being led astray by looking at one informative locus.
But chance may not be what is at work here. Buried in the discussion of the paper:
…polypeptides from several Y-chromosome genes act as male-specific minor histocompatibility (H-Y) antigens that can elicit a maternal immune response during gestation. Such effects could be important drivers of secondary recurrent miscarriages30 and might play a role in the fraternal birth order effect of male sexual orientation.31 Interestingly, all three genes with potentially functional missense differences between the Neandertal and modern humans sequences are H-Y genes, including KDM5D, the first H-Y gene characterized…It is tempting to speculate that some of these mutations might have led to genetic incompatibilities between modern humans and Neandertals and to the consequent loss of Neandertal Y chromosomes in modern human populations. Indeed, reduced fertility or viability of hybrid offspring with Neandertal Y chromosomes is fully consistent with Haldane’s rule, which states that “when in the [first generation] offspring of two different animal races one sex is absent, rare, or sterile, that sex is the [heterogametic] sex.”
The origin of species is obviously one of the founding questions which arose with the emergence of evolutionary biology. Haldane’s rule dates to the 1920s. In mammals the heterogametic sex are males, so these the hybrids which will be selected against (or, they may be sterile). There’s been a lot of research of late on why Neanderthals went extinct, and whether there were speciation barriers in keeping with the biological species concept between our two lineages. This result suggests that there is going to be interesting stuffed coming out of the population genomics of ancient hominins in the near future….