Simulation of model overfit in variance explained with genetic data

Two recent papers, and an author response to prior commentary, addressing the genetic architecture of human temperament and character claimed that “The identified SNPs explained nearly all the heritability expected”. The authors’ method for estimating heritability may be summarized as: Step 1: Pre-select SNPs on the basis of GWAS p<0.01 in the target sample. Step 2: Enter target sample genotypes (the pre-selected SNPs from Step 1) and phenotypes into an unsupervised machine learning algorithm (Phenotype-Genotype Many-to-Many Relations Analysis, PGMRA) for further reduction of the set of SNPs. Step 3: Test the sum score of the SNPs identified from Step 2, weighted by the GWAS regression weights estimated in Step 1, within the same target sample. The authors interpreted the linear regression model R2 obtained from Step 3 as a measure of successfully identified heritability. Regardless of the method applied to select SNPs in Step 2, the combination of Steps 1 and 3, as described, causes inflation of the estimated effect size. The extent of this inflation is demonstrated here, where random SNP selection and polygenic scoring from simulated random data recovered effect sizes similar to those reported in the original empirical papers.