Flexible mixture model approaches that accommodate footprint size variability for robust detection of balancing selection

Long-term balancing selection typically leaves narrow footprints of increased genetic diversity, and therefore most detection approaches only achieve optimal performances when sufficiently small genomic regions (i.e., windows) are examined. Such methods are sensitive to window sizes and suffer substantial losses in power when windows are large. This issue creates a tradeoff between noise and power in empirical applications. Here, we employ mixture models to construct a set of five composite likelihood ratio test statistics, which we collectively termBstatistics. These statistics are agnostic to window sizes and can operate on diverse forms of input data. Through simulations, we show that they exhibit comparable power to the best-performing current methods, and retain substantially high power regardless of window sizes. They also display considerable robustness to high mutation rates and uneven recombination landscapes, as well as an array of other common confounding scenarios. Moreover, we applied a specific version of theBstatistics, termedB2, to a human population-genomic dataset and recovered many top candidates from prior studies, including the then-uncharacterizedSTPG2andCCDC169-SOHLH2, both of which are related to gamete functions. We further appliedB2on a bonobo population-genomic dataset. In addition to theMHC-DQgenes, we uncovered several novel candidate genes, such asKLRD1, involved in viral defense, andSCN9A, associated with pain perception. Finally, we show that our methods can be extended to account for multi-allelic balancing selection, and integrated the set of statistics into open-source software namedBalLeRMixfor future applications by the scientific community.