Sanjana T.
Labs don't sample variants randomly.
Core trick
Applies inverse-propensity weighting to predictor correlations on non-randomly sampled deep-mutational-scan variants, recovering the true correlation within 0.009.
Tags
Description
Labs don't sample variants randomly. They measure hotspots, clinically observed positions, the ones the models argue about, and then compute a predictor's correlation on that set and treat it as that predictor's skill. We show that number is badly biased. On NUDT15 (Suiter et al. 2020), a non-random variant set understates a predictor's true correlation by 0.29 Spearman. Inverse-propensity weighting recovers it to within 0.009, a thirty-fold correction, verified to machine precision.
We also settled the question we started with. We hypothesized that measuring where predictors disagree would identify the best predictor faster than random sampling. We pre-registered the test, committed it before writing the code, and it failed. We sharpened it, pre-registered again, and it failed again, because modern unsupervised predictors, benchmarked across ProteinGym, are statistically distinguishable but practically identical on any given assay. This corroborates Fawzy & Marsh (2024), who found per-gene predictor rankings largely uninformative. We add that the meaningful unit is protein × assay.
So: if you've measured non-randomly, here's your correction. If you're choosing what to measure next, randomize.
README excerpt
# Predictor Arena A CPU-only tool that answers two questions for a scientist studying **one specific protein and one specific assay**: 1. **Which variant-effect predictor should I trust for *my* protein and *my* assay** — judged against my measured variants, with bootstrapped confidence, and reporting *"indistinguishable"* honestly rather than inventing a winner. 2. **Which variants should I measure next** — a 96-well plate selected where predictors most disagree, because that is where my assay carries the most information. No GPU, no model weights, no inference. All predictor scores are downloaded **precomputed** from ProteinGym v1.3. If you see a `pip install torch` here, something has gone wrong. ## 60-second quickstart ```bash git clone <repo> predictor_arena cd predictor_arena python3 -m venv .venv && . .venv/bin/activate pip install -r requirements.txt python arena/download.py # pulls only the 12-assay panel (~228 MB, not the 1.9 GB zip) make demo # regenerates the headline figures ``` > **macOS `~/Downloads` warning.** `~/Downloads` (and `~/Documents`, `~/Desktop`) are TCC-protected. > Shell- and Python-launched processes can be denied access to files there, breaking the pipeline > with `Operation not permitted`. Clone somewhere else (e.g. `~/predictor_arena`), or grant your > terminal **Full Disk Access**. See `DATA_NOTES.md`. ## The two deliverables (they are not the same thing) - **The finding** (submis