Akshay
AI-predicted protein interactomes from AlphaFold-Multimer are now published as catalogues of "high-confidence" complexes and mined for drug targets.
Core trick
Applies conformal FDR control to AlphaFold-Multimer ipTM scores, then checks surviving edges against DepMap co-essentiality data withheld from the structural model.
Description
AI-predicted protein interactomes from AlphaFold-Multimer are now published as catalogues of "high-confidence" complexes and mined for drug targets. But the confidence score they rank on (interface predicted-TM) is overconfident, and the false-discovery rates attached to it are measured on balanced benchmarks with roughly equal real and fake pairs. A real interactome is the opposite: genuine interactions are rare among the thousands of pairs tested, and that rarity is exactly what makes the true error rate dramatically greater than the stated one.The cost of error is real. A target that isn't there sends a drug program down a dead end.
I re-audited the CM4AI cell map (Schaffer et al., Nature 2025) with conformal FDR control, which produces honest error rates without assuming the confidence score is calibrated, then refereed the result against DepMap co-essentiality, a functional signal the structural model never saw, behind a strict anti-data-leakage firewall. The headline finding: a benchmark-tuned "10% error" cutoff actually admits 90% false discoveries as interactions grow rare, while conformal control stays bounded. Under it, 35 of 161 (22%) of the paper's own high-confidence edges fail, and the removed edges are significantly depleted of independent co-essentiality support (p=0.016). I then nominated KANSL3 as a missing member of the leukemia-associated MLL1-WDR5 complex, corroborated by co-essentiality and an independent Boltz-2 structure.
The result is a reusable trust layer that flags which AI-predicted complexes survive honest error control before anyone acts on them. It works only when a release ships the negative controls to calibrate against, which is why CM4AI could be audited and most current deposits cannot.
realakshayk1/emperors-interactome★ 00 forksJupyter NotebookNOASSERTIONCreated Jul 9, 2026 · last push Aug 30, 2026
README excerpt
# The Emperor's Interactome ### When can you audit an AI protein interactome? A distribution-free reliability audit of a published AlphaFold-Multimer map, refereed by held-out biology. --- ## Abstract AI-predicted protein interactomes now label thousands of pairs "high-confidence," but those error rates are estimated on class-balanced benchmarks and inflate sharply in the real regime of a genome-scale screen, where true interactions are rare. The **CM4AI** multimodal cell map (Schaffer et al., *Nature* 2025, Ideker lab; NIH Bridge2AI *Cell Maps for AI* project, contact PI Trey Ideker / co-PI Nevan Krogan, Gladstone Institutes) is among the few AI interactomes that can be audited externally, because it ships a raw confidence axis (AlphaFold-Multimer ipTM) with a native random-pair null — the exchangeable calibration set that distribution-free error control requires. Applying **conformal false-discovery-rate control** to the map's published high-confidence tier, refereed by **DepMap co-essentiality** under a strict purity firewall, **35 of 161 (22%) high-confidence edges fail honest FDR control at q=0.10**, and certified edges are **41% co-essential versus 17% for dropped edges** (permutation p=0.016). We then interrogate the guarantee itself — measuring the exchangeability violation, showing where the marginal guarantee breaks, and reporting the strongly-certified core as robust to violations far larger than the one present — and validate the certified set against orthogo