Chromosomal sex determination is phylogenetically widespread, having arisen independently in many lineages. Decades of theoretical work provide predictions about sex chromosome differentiation that are well supported by observations in both XY and ZW systems. However, the phylogenetic scope of previous work gives us a limited understanding of the pace of sex chromosome gain and loss and why Y or W chromosomes are more often lost in some lineages than others, creating XO or ZO systems. To gain phylogenetic breadth we therefore assembled a database of 4724 beetle species' karyotypes and found substantial variation in sex chromosome systems. We used the data to estimate rates of Y chromosome gain and loss across a phylogeny of 1126 taxa estimated from seven genes. Contrary to our initial expectations, we find that highly degenerated Y chromosomes of many members of the suborder Polyphaga are rarely lost, and that cases of Y chromosome loss are strongly associated with chiasmatic segregation during male meiosis. We propose the "fragile Y" hypothesis, that recurrent selection to reduce recombination between the X and Y chromosome leads to the evolution of a small pseudoautosomal region (PAR), which, in taxa that require XY chiasmata for proper segregation during meiosis, increases the probability of aneuploid gamete production, with Y chromosome loss. This hypothesis predicts that taxa that evolve achiasmatic segregation during male meiosis will rarely lose the Y chromosome. We discuss data from mammals, which are consistent with our prediction.