RECALL
RECALL — a pre-signature coverage agent that detects clinically important omissions in ambient-scribe clinical notes.
Core trick
A grounded entailment judge checks atomic facts from the transcript and FHIR against the note, and an injection harness with known deletions measures recall.
Description
RECALL — a pre-signature coverage agent that detects clinically important omissions in ambient-scribe clinical notes. While ambient documentation has largely solved hallucination, ~76% of its errors are omissions — content said in the visit that silently never reaches the note. RECALL audits every note before signature via an orchestrator–worker pipeline of separate agentic calls: (1) the transcript and encounter FHIR R4 resources are decomposed into atomic, typed clinical facts, each carrying provenance (verbatim quote or FHIR reference); (2) a grounded entailment judge marks each fact present/partial/absent against the note text alone, with verbatim evidence spans; (3) absent facts are classified for expectation and clinical severity; (4) a relevance filter surfaces only expected, safety-critical/major omissions, ranked, with an evaluator–optimizer patch loop that proposes minimal, evidence-grounded note diffs checked by an independent verifier. Because no ground truth exists for omissions, we manufacture it: an injection harness deletes known-present facts from gold notes (confirm-absent QC), yielding a fixed 69-deletion answer key plus 25 untouched false-positive controls — achieving 100% detection recall, a clean-note flag rate of 0.96/note after filtering (1.56 raw upper bound), 0.34% collateral instability, and zero invented facts in a source-vetted multi-agent audit. A stretch demo closes the loop by structuring transcript-only facts (e.g., an uncoded allergy) into FHIR resources and, with mandatory clinician approval, writing them to a sandbox HAPI server (not finished in time)
README excerpt
# RECALL Detects clinically important omissions in AI-generated clinical notes using the visit transcript and FHIR context. Hackathon project for the [Synthetic Ambient FHIR Encounters](synthetic-ambient-fhir-25/) dataset (25 synthetic encounters). ## Setup ```bash python -m venv .venv && source .venv/bin/activate pip install anthropic echo "ANTHROPIC_API_KEY=..." > .env ``` ## Run ```bash python run_checkpoint0.py # baseline notes python run_checkpoint1.py # extract facts + check presence python run_checkpoint2.py # injection eval python run_checkpoint3.py # severity + surfacing python run_checkpoint4.py # patch + verify python run_checkpoint6.py # FHIR writeback demo ``` Results are written to `checkpoint_*.md`. See `TODOS.md` for status.