Gene identity, not variant effect, dominates ClinVar benchmarks of missense pathogenicity predictors
TL;DR
It is found that this test to judge whether a computer program predicts whether a genetic variant causes disease is easier to pass than it looks, and about half the advantage held by programs trained on clinical data disappears once the gene pattern is removed.
Abstract
Missense pathogenicity predictors are routinely benchmarked against ClinVar, whose labels are strongly structured by gene: genes under diagnostic scrutiny accumulate pathogenic submissions while incidentally sequenced genes accumulate benign ones. We asked how much of a benchmark score this structure alone can produce. On 197,904 ClinVar missense variants validated against UniProt canonical sequences, a null model using no variant-level information, scoring each variant only by the pathogenic fraction of its own gene, reaches an area under the receiver operating characteristic curve (AUROC) of 0.921 under a random 10-fold split. On a common intersection of 169,989 variants, four current predictors exceed it by only 0.036 to 0.044. The inflation is not uniform, so it does not cancel when predictors are compared: under within-gene evaluation the ranking inverts, AlphaMissense rising from third to first and gMVP falling to third (p < 0.0001). The inversion survives removal of ceiling genes and replicates on an independently curated benchmark. Because both rankings derive from the same ClinVar labels, we arbitrated between them using data with no gene-level structure: agreement with 47 human deep mutational scanning assays matches the within-gene ranking and inverts the conventional one (p = 0.027, 0.0023). Across twenty-two dbNSFP predictors scored on one common intersection of 112,248 variants, with each tool’s exposure to clinical labels registered before any score was extracted, predictors never trained on such labels sit 0.051 AUROC behind supervised ones globally but only 0.026 behind within genes (difference +0.025 [+0.023, +0.027], p < 0.0001). Leave-one-out correction, the standard remedy, is worth 0.002 AUROC. Much of ClinVar benchmark performance reflects gene identity rather than variant effect, and the distortion changes which predictor a benchmark ranks first, in a direction experimental data contradicts. We release genenull, a single-file implementation, so reporting this baseline costs one function call. Author summary When a computer program predicts whether a genetic variant causes disease, we judge it by testing it against ClinVar, a public archive of variants clinicians have already interpreted. We found that this test is easier to pass than it looks. Some genes appear in ClinVar because they are suspected of causing disease, so most of their recorded variants are harmful; others are sequenced incidentally, so most of theirs are harmless. A program that knows nothing about a variant except which gene it sits in can exploit that pattern, and scores almost as well as the best tools available. This matters beyond a single number. When we removed the gene pattern and ranked variants inside a single gene, the order changed: the tool that looked best became worst, and the one that looked worst became best. Laboratory experiments that measure the effect of every possible variant in a protein agree with the new order, not the old one. Across twenty-two prediction tools, about half the advantage held by programs trained on clinical data disappears once the gene pattern is removed. We release software so anyone can measure this baseline in one line of code.
