Numici
This system verifies whether claims in AI-generated research reports faithfully represent their cited sources.
Core trick
Matches each cited claim to its source text and flags mismatched authors, years, numbers, and inverted findings against the retrieved document.
Description
This system verifies whether claims in AI-generated research reports faithfully represent their cited sources. It is designed around the "AI-assisted, human-verified" principle:
README excerpt
# Numici — Citation Faithfulness Verification Verifies whether the claims in a research report — **AI-assisted or human-written** — **faithfully represent their cited sources**, and flags the specific way each claim departs from its source. Built for analysts and researchers who put their name on a report and need every cited claim to hold up. (AI synthesis makes these errors constantly — but so do humans; the tool doesn't care who wrote the draft.) > **Why this matters.** LLM research output looks authoritative but routinely > misrepresents sources: fabricated citations, attributions to the wrong > author/year/filing, inverted findings, fabricated or miscalculated numbers, > stripped caveats, and over-generalized conclusions. Numici checks each claim > against the source it cites and tells the analyst *whether* it holds up and > *how* it fails. --- ## What it does For every (claim, citation) pair in a report, Numici runs a two-stage pipeline: 1. **Extraction + citation resolution** (`claude-haiku-4-5`): pull the (claim, citation) pairs, resolve the cited source URL, and record a resolution status — `resolved` / `404` / `paywalled` / `blocked` / `timeout` / `malformed`. (`blocked` = the host refused the automated fetch, e.g. legal databases that block bots.) 2. **Faithfulness judgment** (`claude-sonnet-4-6`, two sequential prompts — never combined): - **Prompt A — citation match:** is this citation the right source for this claim? - **Prompt B — fai