Dima
Even top ML PhDs cap out at a 41.4% replication score after 48 hours of dedicated effort per paper (PaperBench, OpenAI, 2025).
Core trick
Four agents statically cross-check a paper's claims against its code and data, requiring two agents to agree before a finding is reported.
Description
Even top ML PhDs cap out at a 41.4% replication score after 48 hours of dedicated effort per paper (PaperBench, OpenAI, 2025). Full replication is too slow and too hard, even for experts — which is exactly why static diagnosis has to be the first line of defense. The bottleneck isn't compute, it's diagnosis. The bug — an augmentation leak, a train-test overlap, a metric mismatch, a missing `model.eval()` — was almost always a static read away.
RunItBack does the static read. Point it at a paper, a repo, and some data; four Opus 4.7 agents (Paper Analyst, Code & Data Auditor, Validator, Reviewer) running on Claude Managed Agents return a diagnostic report with a verdict, claim-by-claim verification, severity-ranked findings, and unified-diff fixes — in minutes, before any GPU starts.
Opus 4.7 ingests PDFs natively (tables, figures, equations — no OCR, no `pdftotext`), carries the full reproducibility-failure taxonomy across 60-turn tool-using sessions, and enforces a ≥ 2-agent cross-check rule that surfaces genuine disagreements instead of hallucinating consensus. Managed Agents provides the sandbox, tools, sessions, and streaming on day one — RunItBack adds only the brain.
README excerpt
# RunItBack **Diagnose ML paper reproducibility before you burn a single GPU hour.** RunItBack takes a machine-learning paper and its companion code repository and returns a diagnostic report telling you whether the code actually implements what the paper claims — with claim-by-claim verification, severity-ranked findings, and suggested fixes as unified diffs. It is a tool for the kind of reproduction work where the failure modes are subtle and the surface area is large: four Claude Opus 4.7 agents running on Claude Managed Agents do the close reading, auditing, runtime validation, and cross-checking that one researcher would otherwise spend a week on. The diagnosis arrives in minutes; the experiments you actually want to run start sooner. | | | |---|---| | **Demo video** | https://drive.google.com/file/d/1c0yJFpPJjAiKCjmJxmjYwY-jLp1eLEur/view?usp=sharing| --- ## Why this exists **Even top ML PhDs cap out at a 41.4% replication score after 48 hours of dedicated effort per paper** (PaperBench, OpenAI, 2025). Manually checking a single replication attempt can take tens of expert hours. Full reproducibility is, on a per-paper basis, prohibitively expensive — and the gap is not about hardware, it is not about luck, and it is very rarely about needing more compute. It is about **augmentation leaks, train-test overlap, config drift between paper and code, silent broadcasting bugs, missing seed discipline, evaluation metrics that don't match the one the paper reported, and und