Paper Trail
Paper Trail is a Claude-powered agent and web dashboard that investigates why an ML paper's published result fails to reproduce from its public repo.
Core trick
A Claude agent runs discriminating checks via tool use against a paper's repo, applies a minimal patch, re-runs evals, and opens a GitHub PR.
Description
Paper Trail is a Claude-powered agent and web dashboard that investigates why an ML paper's published result fails to reproduce from its public repo. It identifies the root cause, applies the minimal fix, re-runs the evaluations, and opens a real GitHub PR with an evidence-backed scientific dossier. It targets the boring, expensive layer of the reproducibility crisis: data leakage, train/test contamination, label-aware preprocessing, and metric-implementation drift; the failures that quietly inflate headline numbers and rarely get caught in review.
The product runs in two modes. Deep Investigation is the autonomous flow: paste a paper URL and a repo URL then watch the agent generate ranked hypotheses, run discriminating checks via tool use, converge on a verdict with cited evidence, patch the code, and post a before/after metric delta. Quick Check is a chat-style sidebar where a researcher asks targeted, bounded questions ("is this imputation fit on train only?") and gets a verdict (confirmed OR refuted OR unclear) with at least one code citation in under 30 seconds.
Together they reframe the agent from "autonomous scientist" to "verification intern"; one a researcher can actually trust to do the unglamorous audit work.
README excerpt
# Paper Trail (Reproducibility Forensics) > **A verification intern for research engineers.** Paste a paper + a GitHub repo. Ask a quick verification question — *"does this split respect patient boundaries?"* — or launch a full investigation — *"why doesn't this paper reproduce?"*. Get back evidence-backed answers, and when the fix is clear, a real GitHub PR with a scientific audit dossier and an independent peer-review pass. Built for the **"Built with Opus 4.7" Claude Code hackathon** (Cerebral Valley, Apr 21–26, 2026).  --- ## Why this exists A huge amount of ML research breaks at the same point: a paper claim exists, the public repo is incomplete or drifted, and nobody can quickly tell whether the gap is in the code, the data, the config, or the paper's prose. Time and GPU hours die there, and the scientific record loses credibility. **Paper Trail** is a Claude Code agent that does what senior research engineers usually delegate to PhD students: read the paper, read the repo, form ranked hypotheses about why the numbers don't match, design and run small discriminating checks, and — when the root cause is clear — write the minimal fix and produce a reviewable audit trail. --- ## Two modes, one composer ### Deep Investigation Paste a paper URL + a repo (GitHub URL, `owner/repo` slug, or local path — the backend auto-clones). The chat shows four live blocks: - **Hypothesis Board** — ranked candidate failure causes with