PertEMA
PertEMA is a post-hoc, model-agnostic reliability layer for single-cell perturbation-effect predictors.
Core trick
A gradient-boosted meta-model predicts each perturbation prediction's error from prediction-time features, then isotonic and conformal calibration yield abstention intervals.
Tags
Description
PertEMA is a post-hoc, model-agnostic reliability layer for single-cell perturbation-effect predictors. Deep models that predict how a CRISPR perturbation reshapes gene expression are used to nominate experimental targets, but on unseen perturbations they rarely beat a trivial mean baseline. Instead of building a better predictor, I ask which individual predictions can be trusted: a gradient-boosted-tree meta-model estimates each prediction's error from prediction-time features only, with isotonic calibration and split-conformal intervals.
On the genome-scale Gladstone CD4 T cell Perturb-seq screen, the reliability score ranks realized error above the effect-magnitude and training-similarity heuristics, calibrates to near-nominal coverage, and builds a validation shortlist that is measurably more reproducible than ranking by effect size. I also characterized where reliability cannot help: per-instance model routing is infeasible, because candidate predictors co-fail on the same hard, noise-dominated perturbations. I quantified this with a measured noise ceiling and a break-even condition, and I pre-registered a routing test on an independent screen that failed exactly as predicted.
Why it matters: it gives perturbation-prediction users a calibrated, honest abstention signal on real primary data, plus a transferable, reproducible finding about the limits of per-instance model selection, shipped as a self-hostable tool and an open reliability benchmark.
README excerpt
# PertEMA **Per**turbation **E**rror **M**eta-**A**ssessment. A study of when per-prediction reliability can and cannot be exploited in single-cell perturbation prediction, with a post-hoc reliability estimator and a self-hostable demo so others can reproduce and apply the finding. License: MIT. ## The question and the finding Deep models that predict how a genetic perturbation reshapes single-cell gene expression are used to nominate experimental targets, yet on unseen perturbations they frequently fail to beat a trivial mean baseline. Rather than build a better predictor, this project asks which individual predictions can be trusted, and what a per-prediction reliability signal can and cannot be used for. On the public genome-scale Gladstone CD4 T cell Perturb-seq screen, we find: - No predictor in the roster beats the per-condition mean baseline. Every bootstrap confidence interval on error-relative-to-the-mean sits above it (Fig. 1). - A post-hoc reliability estimator, trained on out-of-fold errors from prediction-time features only, ranks realized error above the effect-magnitude and similarity heuristics and calibrates to near-nominal coverage, so it supports selective abstention. - Ranking a validation shortlist by reliability is measurably more reproducible than ranking by effect size, the practical payoff for a lab. - Per-instance model routing is infeasible: candidate predictors co-fail on the same hard, noise-dominated perturbations. We quantify thi