Skip to content

falsify

Teams ship AI accuracy claims that evaporate under scrutiny not from malice, but because the claim was never structured to be falsifiable.

Code Video Cerebral Valley

Core trick

Seals metric, threshold, and dataset with SHA-256 before the run, then exits 3 in CI when a later edit breaks the hash.

Tags

dev toolsscience

Description

Teams ship AI accuracy claims that evaporate under scrutiny not from malice, but because the claim was never structured to be falsifiable. A "94% accuracy" number becomes 71% in production and nobody can tell whether the model drifted, the metric got relaxed, or the threshold moved after seeing the data. Falsify is a CLI that forces pre registration: you cryptographically seal the metric, threshold, and dataset with SHA-256 before the experiment runs. Post hoc edits break the hash. CI exits 3. The audit trail writes itself. Deterministic exit codes are the API — 0 pass / 10 fail / 3 tampered. CI gates on the code, humans read the trail. The claim either survives contact with the data, or it didn't happen. Built entirely with Claude Opus 4.7: 5 skills, 2 subagents, 3 slash commands, 1 MCP server. 514 tests passing.
studio-11-co/falsify★ 83 forksPythonMITCreated Apr 24, 2026 · last push Oct 23, 2026

README excerpt

<a href="https://falsify.dev"><img src="brand/hero.png" alt="falsify: fix the pass bar before the test runs. Anyone can check it later." width="100%"></a>

[![CI](https://img.shields.io/github/actions/workflow/status/studio-11-co/falsify/falsify.yml?branch=main&label=CI&style=flat-square)](https://github.com/studio-11-co/falsify/actions/workflows/falsify.yml)
[![Multi-lang Conformance](https://img.shields.io/github/actions/workflow/status/studio-11-co/falsify/multi-lang-conformance.yml?branch=main&label=conformance%20%C2%B7%204%20languages&style=flat-square)](https://github.com/studio-11-co/falsify/actions/workflows/multi-lang-conformance.yml)
[![tests](https://img.shields.io/badge/tests-646%20passing-2ea44f?style=flat-square)](https://github.com/studio-11-co/falsify/actions/workflows/falsify.yml)
[![vectors](https://img.shields.io/badge/vectors-21%20positive%20%C2%B7%2020%20reject%20%C2%B7%2092%20edge-2ea44f?style=flat-square)](spec/test-vectors/)
[![OpenSSF Best Practices](https://img.shields.io/cii/level/13539?label=OpenSSF%20best%20practices&style=flat-square)](https://www.bestpractices.dev/projects/13539)
[![honesty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/studio-11-co/falsify/main/.falsify/badge.json&label=honesty&style=flat-square)](claims/self/)
<br>
[![PyPI](https://img.shields.io/pypi/v/falsify?label=pypi%20falsify&color=2ea44f&style=flat-square)](https://pypi.org/project/falsify/)
[![npm](https://img.shields.io/npm/v/falsify-js?label=n

Team