CoEvoLve
CoEvoLve — A Living Benchmark That Evolves Faster Than Any Model Can Game It.
Core trick
Dual-archive coevolution: Gemini examiners mutate causal reasoning tasks to maximize separation between solver strategies, which evolve in return.
Description
CoEvoLve — A Living Benchmark That Evolves Faster Than Any Model Can Game It.
Static benchmarks leak. CoEvoLve solves this with a coevolutionary system where the benchmark and the solvers evolve together in an open-ended arms race.Two populations of Gemini agents co-evolve simultaneously: Examiners generate causal reasoning tasks and evolve toward maximum discriminative power — tasks that separate strong solvers from weak ones. Solvers attempt those tasks and evolve better reasoning strategies. .After 500 generations, the system independently discovered that graphs combining confounders, mediators, and colliders are maximally discriminative — a finding that matches the theoretical causal inference literature, without being programmed. The benchmark found the structure of LLM failure automatically.Built on ShinkaEvolve (Sakana AI, Apache 2.0) extended with dual-archive coevolution, Gemini 3 Pro for examiner mutations, Gemini 3 Flash for solver evaluation.
README excerpt
# CoEvoLve > A living benchmark that evolves faster than any model can game it. **Gemini 3 Paris Hackathon 2026** · Built with Gemini 3 Pro + Flash · Google Antigravity --- ## The problem with benchmarks Static benchmarks leak. Once a model trains on the internet, it trains on the benchmark too. The solution isn't a bigger benchmark — it's a benchmark that *evolves*. ## What CoEvoLve does Two populations of Gemini agents co-evolve in an open-ended arms race: - **Examiners** generate causal reasoning tasks, evolving toward maximum discriminative power — tasks that separate strong solvers from weak ones - **Solvers** attempt those tasks, evolving better reasoning strategies Neither population has a fixed target. The examiner's fitness depends on the current solver population. The solver's fitness depends on the current examiner population. This is the Red Queen dynamic applied to AI benchmarking. A human-in-the-loop layer allows optional steering: rate tasks on the Pareto frontier to bias the evolutionary search toward scientifically interesting regions. ## Why causal reasoning LLMs have a well-documented, measurable failure mode on causal reasoning: ``` Graph: U → X, U → Y, X → Y (U is hidden confounder) P(Y=1 | X=1) ≈ 0.72 ← what baseline LLMs report P(Y=1 | do(X=1)) ≈ 0.55 ← correct causal answer Error: +0.17, consistently, across model families ``` Ground truth is computed by exact do-calculus (pgmpy). No LLM judge. No hallucination risk in the ev