Grounded Genomics
I built a benchmark and a prompt-level "closed-world grounding" layer that measurably reduces hallucination when an LLM writes investigation summaries for genetic variants of uncertain significance (VUS).
Core trick
Injects only verified facts plus a non-fabrication contract into the prompt, then scores outputs with four detectors tied to source-data properties.
Description
I built a benchmark and a prompt-level "closed-world grounding" layer that measurably
reduces hallucination when an LLM writes investigation summaries for genetic variants
of uncertain significance (VUS).
The problem: AI tools are increasingly used to summarize evidence for VUS, and when
the evidence is thin they fabricate — inventing HGVS coding notation, wrong evidence
values, misattributed citations, or clinical Pathogenic/Likely-Pathogenic calls the
source pipeline never made. Any of these can push a curator, or a family, toward the
wrong conclusion. This is personal for me: it comes out of my family's connection to
the RASopathy world.
The method: over 163 curated Tier-1 RASopathy variants, I run the same model on the
same variant with two prompts — an unconstrained baseline and a grounded arm that
injects only verified facts plus an explicit non-fabrication contract. Four
dataset-anchored detectors then score every output, each tied to a verifiable property
of the source data (e.g. the source contains zero HGVS c. notation, so any c. notation
is fabrication by definition).
The result: grounding raised the clean-output rate from 0.42 to 0.91 raw / 1.00
adjudicated, and calibration from 0.43 to 1.00, cutting mean fabrications per variant
from 1.50 to ~0.11. I also report an honest raw-vs-adjudicated finding: my high-recall
clinical detector produced 17 false positives in the grounded arm (hedged/negated
language), so I publish both numbers rather than silently tuning the detector — the gap
quantifies the detector's own calibration.
Everything is reproducible: the repo rebuilds the published results from saved model
outputs with no API key, and an interactive Space lets you see each fabrication
highlighted inline, grounded vs ungrounded, for any variant.
README excerpt
# Closed-World Grounding for AI-Assisted RASopathy Variant Classification A prompt-level **closed-world constraint layer** plus four dataset-anchored **fabrication detectors** that measurably reduce hallucination when an LLM writes variant-investigation summaries — benchmarked over **163 Tier-1 RASopathy variants** on `claude-sonnet-4-5` at temperature 0.0. **Headline result:** constraining the model to a closed world of verified facts raises the clean-output rate from **0.42 → 0.91** (raw detector) / **1.00** (adjudicated) and calibration from **0.43 → 1.00**, while cutting mean fabrications per variant from **1.50 → 0.11**.  --- ## Why this project This work comes from lived experience. A close personal connection to the RASopathy world — and to a specific **SOS1** variant of uncertain significance — is what pulled me toward the variant-interpretation problem. When you sit with a VUS report that matters to someone you love, you feel the exact failure mode this benchmark targets: a tool that sounds confident and clinical when the evidence underneath does not support a call. RASopathies (Noonan syndrome and related conditions) are driven by variants in the RAS/MAPK pathway — `SOS1`, `PTPN11`, `RAF1`, `BRAF`, `MAP2K1/2`, `RIT1`, `SHOC2`, `SOS2`, `HRAS`. Most newly observed missense changes land in the enormous grey zone of "uncertain significance." LLMs are increasi