AGET-Bench
AGET-Bench — a benchmark for governance-overlay agent behavior.
Core trick
Runs generated code against a deterministic rule checker at 5, 10, and 20 rules to measure compliance without self-reported scores.
Description
AGET-Bench — a benchmark for governance-overlay agent behavior. This public slice asks: does a coding model's rule-compliance degrade as more rules are presented to it? Compliance is computed by a deterministic rule-checker executed against the model's generated code — never self-reported. Result: density-invariant for Opus 4.8 — compliance holds across a 4× range of rule count (5 / 10 / 20 rules: 80% / 75% / 76%, 95% CIs overlap). Build Day is the public release of this ongoing benchmark, with transparent handling of degenerate (prose-only) runs.
README excerpt
# AGET-Bench — Instruction-Density Ladder (public slice)
A slice of AGET-Bench: does an LLM's compliance with explicit coding rules degrade as the
number of presented rules rises? Built on **Claude Build Day, 2026-06-13**, orchestrated via
the AGET governed workflow (gated plan + rubric the run is scored against).
## Built on Build Day vs vendored prior work
| Path | Origin | Status |
|------|--------|--------|
| `original/` | authored 2026-06-13 | Build Day contribution |
| `vendor/` | poc/vocab-spec-compliance (Feb 2026) | prior work, headered; included only so the slice runs standalone |
`original/` = the density-ladder methodology (subsetting a fixed-mix rule corpus), the
isolated-sandbox runner, the presented-vs-unpresented density-aware scorer, the clean
pre-registration, and the opus-4.8 results produced 2026-06-13.
## Findings & limits (honest)
- Shipped conditions: **d05: 16 scoreable + 4 prose-only (n=20) | d10: 20 scoreable + 0 prose-only (n=20) | d20: 20 scoreable + 0 prose-only (n=20)**. ("prose-only" = the subject answered in prose with no extractable
code block; the scorer excludes these from the pass-rate denominator and reports the count.)
- **Result — presented-rule compliance is density-invariant** (scored 2026-06-13 via
`score_density_ladder.py`; PRESENTED strict %, Wilson 95% CI):
| density | n | PRESENTED strict % [95% CI] |
|--------:|--:|-----------------------------|
| d05 (5 rules) | 16 | 80.0 [70.0, 87.3] |
| d10 (10 rules) | 20 |