Provenance Gate (solo team)
I investigated a failure mode of agentic science: as an agent forks approaches, revises an upstream step, and re-runs, a final result can quietly come to rest on stale or version-conflicting inputs.
Core trick
Walks the computation lineage graph deterministically to flag a result whose upstream inputs include two live versions of one artifact.
Description
I investigated a failure mode of agentic science: as an agent forks approaches, revises an upstream step, and re-runs, a final result can quietly come to rest on stale or version-conflicting inputs. The conflict lives in the provenance, not the prose — so reading the output, a human or an LLM reviewer may have nothing to go on.
To study it in Claude Science I built two instruments: a deterministic, read-only provenance gate that computes two structural verdicts per computation (a stale input; a lineage that reconverges on two live versions of one artifact), and a first-prototype harness that runs a scenario unattended and captures each rollout for grading.
The finding: across 24 unattended rollouts of one PBMC single-cell scenario, 12 shipped a version-mixed figure and 6 shipped it with no warning anywhere in the conversation. Installing the gate as a skill wasn't enough for an agent to use it on its own; one line of standing policy was, and the agent then caught the conflict and stopped to ask. Yet handed the gate's flag directly, an agent still rationalized a real conflict as "benign" — which is exactly why the check must be deterministic: the verdict is read from the lineage, not the model's judgment.
Why it matters: fast agentic science is only trustworthy if results trace to consistent inputs — a structural fact a deterministic check can hold even when the prose can't, and a natural candidate for a verifiable training signal, not just a review. Design rationale, limitations, and full rollout results are in the repo.
README excerpt
# Provenance Gate *A deterministic, read-only trust gate for agentic science over Claude Science (CS). It flags a result quietly built on stale or version-conflicting upstream data — a conflict that lives in the provenance graph, not the prose.*  *The gate inside Claude Science. The agent renders the cockpit as a read-only artifact; opening it and clicking the flagged cell shows the deterministic `version_mix` verdict, with the conflicting versions named.* ## The problem An agent doing research over many turns forks approaches, revises an upstream step, and re-runs. Along the way a conclusion can come to rest on an old version of a file, or on a report that combined two divergent versions of the same artifact. Reading the final text, an LLM reviewer often has nothing to go on, because the conflict is in the provenance, not the prose. ## What it does The gate reads the CS provenance DAG and computes two verdicts for each computation cell: - `stale_input` — the cell read a version of some artifact that is no longer current. - `version_mix` — the cell's consumed lineage reaches two live versions of the same artifact. Both are structural facts about the graph. No model decides them: an agent can argue a flag is harmless — one did, in testing — but a flag is a fact abou