AgentLab
When a multi-agent pipeline produces a confident wrong answer — which step broke, and how do you know?
Core trick
Localizes quality regressions to specific pipeline steps by scoring each step's traces against drafted, human-approved eval cases.
Description
When a multi-agent pipeline produces a confident wrong answer — which step broke, and how do you know?
AgentLab is an evaluation system for multi-step agent pipelines. It watches production traces, detects quality drift, and does the part nobody else automates: it localizes the regression to the specific step that caused it. From the real failing traces it drafts targeted evaluation cases; a human approves them; the fleet is re-scored against the enlarged suite; the degraded step is rerouted to a healthy agent
README excerpt
# AgentLab **When a multi-agent pipeline produces a confident wrong answer — which step broke, and how do you know?** AgentLab is an evaluation system for multi-step agent pipelines. It watches production traces, detects quality drift, and does the part nobody else automates: it **localizes the regression to the specific step that caused it**. From the real failing traces it drafts targeted evaluation cases; a human approves them; the fleet is re-scored against the enlarged suite; the degraded step is rerouted to a healthy agent. Built on Gemini 3.5 Flash. The demo pipeline genuinely spans three clouds. --- > **Built for the post-I/O '26 Gemini hackathon.** The headline is **step-level fault localization** — credit assignment in a multi-agent pipeline, attributing a downstream wrong answer to the upstream step responsible. The loop around it — detect → localize → generate evals → *human approves* → re-score → reroute — is how that localization becomes a measurable repair, live, inside a three-minute demo. --- ## The Hard Problem: Which Step Broke? A multi-step agent pipeline returns a bad answer. Three steps touched it — in a larger system, forty. Which one caused the regression? This is a **credit-assignment problem**, and it is not solved. The agent that emits the visibly-wrong output is frequently not the one that broke: a quietly degraded *upstream* step hands plausible-but-wrong intermediate state to a healthy *downstream* step, which faithfully turns it into a