CBMR Multimodal

**What we investigated.** ML leaderboards present ranked models with narrow error bars, which reads as evidence of predictive strength. But a February 2026 audit of the Therapeutics Data Commons (TDC) ADMET benchmark (Receptor.AI, bioRxiv 2026.02.26.708193) found that most top-ranked entries fail on unavailable code, non-reproducible environments, or data leakage, and that deliberately overfitting a model to the open test set moves it up the rankings. On a benchmark with a fully open test set, climbing and cheating can be the same act. We asked: can an AI agent iteratively improve ADMET models *honestly*, and what does honest iteration reveal about the benchmark itself? **What we built.** An agentic pipeline, driven by Claude Code as a research agent, that does two things in one loop. It **climbs**: an agent proposes model designs, trains them, and selects among them to improve validation performance. And it **audits**: the same loop checks every model and data split for leakage and reports its own results with honest uncertainty. Honesty is enforced structurally, not by good intentions. Four roles are separated so the component that touches the test set cannot influence the one that designs models: a proposer (designs models on validation only), an executor (trains and scores on validation), a deterministic arbiter (promotes only on non-overlapping confidence intervals; keeps simpler models on ties), and a walled auditor (the only component allowed to touch test-set molecules, for leakage checks, and it returns only a summary verdict with no labels). The test set is reachable only by the auditor and by a single final-evaluation step, run once after a model is locked. The iteration loop has no code path to the test set. The human is inside the loop but subject to the same accounting as the agent: repeated testing widens the reported uncertainty, and the tool refuses high-complexity model tiers on endpoints too small to support them. **What we found**, across 10 of the 22 ADMET endpoints: - Honest iteration climbs where there is signal and plateaus where there is not. In 3 of 10 endpoints models measurably separated, all the same physically motivated move (physicochemical descriptors beating substructural fingerprints on descriptor-driven properties). In the other 7 the models were statistically indistinguishable, and the loop reported that instead of manufacturing a winner. - On small endpoints, the leaderboard ranks noise. The test-set bootstrap interval on small imbalanced endpoints is wide enough to span most of the leaderboard, so the fine-grained rankings there are not supported by the sampling error. - A documented leakage bug drifted silently across data versions (patched on two endpoints, still live on a third), and a two-axis leakage check proved necessary: on one endpoint, real duplicates that structural similarity missed were caught only by a desalted-identity check, while on another, a structural false positive was correctly cleared by the identity axis. **Why it matters.** Our contribution is not a better ADMET model. It is a demonstration that an AI agent can do the iterative modeling work without the test-set overfitting that compromises the benchmark, and can audit the benchmark's integrity as it goes, producing results honest about what the data can and cannot support. The same pipeline, run as new models are submitted, is a standing integrity check on a leaderboard rather than another entry gaming it. The full pipeline, the 10-endpoint findings, and the cached data reproduce from a fresh clone on a standard laptop.