The evolutionary response of mating system to heterosis

Alexander Harkness, Yaniv Brandvain, Emma E. Goldberg

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


Isolation allows populations to diverge and to fix different alleles. Deleterious alleles that reach locally high frequencies contribute to genetic load, especially in inbred or selfing populations, in which selection is relaxed. In the event of secondary contact, the recessive portion of the genetic load is masked in the hybrid offspring, producing heterosis. This advantage, only attainable through outcrossing, should favour evolution of greater outcrossing even if inbreeding depression has been purged from the contributing populations. Why, then, are selfing-to-outcrossing transitions not more common? To evaluate the evolutionary response of mating system to heterosis, we model two monomorphic populations of entirely selfing individuals, introduce a modifier allele that increases the rate of outcrossing and investigate whether the heterosis among populations is sufficient for the modifier to invade and fix. We find that the outcrossing mutation invades for many parameter choices, but it rarely fixes unless populations harbour extremely large unique fixed genetic loads. Reversions to outcrossing become more likely as the load becomes more polygenic, or when the modifier appears on a rare background, such as by dispersal of an outcrossing genotype into a selfing population. More often, the outcrossing mutation instead rises to moderate frequency, which allows recombination in hybrids to produce superior haplotypes that can spread without the mutation's further assistance. The transience of heterosis can therefore explain why secondary contact does not commonly yield selfing-to-outcrossing transitions.

Original languageEnglish (US)
Pages (from-to)476-490
Number of pages15
JournalJournal of evolutionary biology
Issue number5
StatePublished - May 2019

Bibliographical note

Publisher Copyright:
© 2019 European Society For Evolutionary Biology. Journal of Evolutionary Biology © 2019 European Society For Evolutionary Biology

Copyright 2020 Elsevier B.V., All rights reserved.


  • heterosis
  • inbreeding
  • mating systems
  • self-fertilization

Fingerprint Dive into the research topics of 'The evolutionary response of mating system to heterosis'. Together they form a unique fingerprint.

Cite this