Garry Kuwanto
Stakeholder feedback for LLM systems is informal: "the classifier feels off on customer reviews," "the assistant is missing the point lately." Translating that into a regression test takes a day of writing benchmarks, ha
Core trick
Decomposes a failure hypothesis into probing dimensions, generates candidate inputs, and keeps only inputs where the two prompt variants diverge.
Tags
Description
Stakeholder feedback for LLM systems is informal: "the classifier feels off on customer reviews," "the assistant is missing the point lately." Translating that into a regression test takes a day of writing benchmarks, hand-checking discrimination, and prompt-tuning toward a moving target.
Hypothesize collapses that loop. Given a hypothesis about a system's failure mode and the system's current prompt, it asks Claude to decompose the hypothesis into probing dimensions, generates targeted candidate inputs, runs both the current system and an alternative, and keeps only the inputs where they meaningfully diverge. The output is a small YAML benchmark designed to discriminate between the two — every test exists for a stated reason.
The same primitive ships as a CLI, a Claude Code skill that turns natural-language complaints into committed regression tests, and an MCP server exposing the workflow to any compatible host.
One emoji-overuse hypothesis: 30 questions tested, 5 discriminating cases, 0 regressions, $0.02.
README excerpt
# Hypothesize Turn informal LLM failure hypotheses into minimum discriminating regression benchmarks. A stakeholder complains. An engineer wants to verify the complaint and produce a targeted regression test in under a few minutes and a few dollars of tokens. Hypothesize is built for that loop: it generates the *minimum set* of inputs that discriminate between a current system and a plausibly-better alternative, given a stated hypothesis. Tests have a stated reason for existing — the hypothesis they prove. Status: 0.1.0 (early development). See `.spec/` for the current plan. ## Install The PyPI distribution name is **`hypothesize-cli`**; the import name and console script are `hypothesize`. **With pip:** ```bash pip install hypothesize-cli hypothesize setup ``` **With uvx (no permanent install):** ```bash uvx --from hypothesize-cli hypothesize setup ``` If you don't have `uv` yet: `pip install uv`. The setup wizard will: - Configure your Anthropic API key (written to `~/.config/hypothesize/.env` with mode `0600`) - Install the Claude Code skill if `claude` is on your PATH - Register the MCP server with Claude Desktop if its config directory is present You can skip any step. Re-run `hypothesize setup` anytime to reconfigure. For CI use: ```bash hypothesize setup \ --non-interactive \ --api-key sk-ant-... \ --skip-claude-code \ --skip-claude-desktop ``` ## Quickstart After setup, try the bundled sarcasm example: ```bash hypothesize run \ --config