Kekule
I agent benchmarks today are static: run agents, score results, manually tweak prompts, repeat.
Core trick
A coordinator rewrites the playbook after each epoch from failure analyses, and oracle tests gate each waypoint of a GitHub issue fix.
Description
I agent benchmarks today are static: run agents, score results, manually tweak prompts, repeat. Researchers spend more time diagnosing failures and hand-tuning strategies than running experiments. The feedback loop between "what went wrong" and "what to try next" is entirely manual -- and it doesn't scale.
Kekule is a Claude Code skill that closes this loop. You define a task set and point kekule-bench at it. A swarm of Claude agents solves tasks, then the system runs its own retrospective: a failure analyst diagnoses why each fix broke, a coordinator distills generic lessons and invents new verification strategies, and the next epoch runs with a rewritten playbook.
The verification engine evolves autonomously. In our first experiment, the coordinator invented 5 oracle strategies that didn't exist at startup. We targeted SWE-bench tasks the current SOTA couldn't solve and resolved 2.
README excerpt
# Kekule > *Named after August Kekule's famous dream of a snake eating its own tail -- the ouroboros that led to the discovery of benzene's ring structure. Just as that moment of lateral thinking revolutionized chemistry, we explore how analogical reasoning and structured exploration can unlock breakthrough capabilities in agent swarms.* A self-improving heterarchical agent swarm that solves real GitHub issues by decomposing work into verified waypoints, with oracle tests that gate each step. **Built for**: [Built with Opus 4.6: a Claude Code hackathon](https://cerebralvalley.ai/) (Cerebral Valley & Anthropic) **Team**: [Ansh Tulsyan](https://github.com/archi-max) & [Jack Armitage](https://github.com/jarmitage) --- ## Results: Beating SOTA on SWE-bench ### Targeting tasks SOTA can't solve We referenced the SOTA results from [openautocoder/live-swe-agent](https://github.com/openautocoder/live-swe-agent) on SWE-bench Verified (75.4% resolve rate). Their evaluation results ([`docs/live-swe-agent-results/eval_result.json`](docs/live-swe-agent-results/eval_result.json)) showed 117 unresolved tasks. We selected a subset of these as our target list (see [`docs/target-tasks.md`](docs/target-tasks.md)). During the hackathon, we ran our perturbation swarm on a handful of these target tasks and solved 2 that the SOTA agent couldn't: | Task | Tests Fixed | How | |------|------------|-----| | `django__django-15022` | 3/3 | Swarm decomposed into root_cause_analyzer + fix_implemen