n = 3, and Proud
Everyone building research agents runs into the same quiet question: once you have one, how should you set it up to do its best work?
Core trick
A blank coding agent drives a research agent in a memory-free loop, compared against an unscaffolded baseline on identical questions with blind scoring.
Description
Everyone building research agents runs into the same quiet question: once you have one, how should you set it up to do its best work? Usually the answer is a hunch. This project turned the hunch into an experiment. A researcher used one AI research agent to study how to make that same agent produce better science — running a blank coding agent as a driver in a closed, self-improving loop, and pitting each loop design against a raw, unscaffolded baseline on word-identical questions, scored blind by an automated reviewer panel and a human domain expert.
Across three rounds, the matured design scored highest on every metric and for every scorer, and beat its predecessor on all three test questions. The effect is modest and the sample is small — three questions, and one open-ended question the baseline won — and the report says so plainly. The lasting contribution is not the winning configuration; it is a reusable, auditable way to measure whether an agent's setup actually helps, rather than asserting that it does, paving the way for individual harness and workflow research to produce better science.
BobtheBuilder0426/The-Metascience-Project★ 00 forksHTMLMITCreated Jul 13, 2026 · last push Jul 13, 2026
README excerpt
# The Metascience Project **Can you use an AI research agent to discover how to make that same agent do better science — and *measure* that it worked?** This repository is the complete, auditable record of a small meta-science project that treats that question as an experiment rather than a hunch. A human operator ran an outer **experiment loop** (hypothesise → test → analyse → repeat) to search for a good inner **agentic loop**: a blank general-purpose coding agent that bootstraps and drives a research agent through a fixed, memory-free cycle. Each candidate loop design was compared against a raw, unscaffolded baseline of the *same* research agent, on word-identical questions, scored blind by two independent judges. **Headline finding (stated honestly).** Across three iterations, the matured loop design (v8) scored highest on every aggregate metric and for every scorer, and improved on its predecessor (v7) on all three test questions. The effect is **modest and the sample is small** (n = 3 questions; one open-ended question was won by the baseline), so we report a **consistent direction, not a powered effect**. The durable contribution is the **reusable, auditable method** for measuring whether an agent's configuration actually helps. --- ## Start here (for a judge) | If you have… | Read this | |---|---| | **2 minutes** | [`submission/summary.md`](submission/summary.md) — 174-word plain-language summary | | **15 minutes** | [`submission/report/report.pdf`](submission/r