Skip to content

Agent Qualification Lab

Agent Qualification Lab helps developers evaluate an AI agent before trusting it with consequential actions.

Code Video Cerebral Valley

Core trick

Records attempted tool calls separately from completed tasks, then compares a frozen retrieval filter against the baseline in a synthetic billing environment.

Tags

securityagentsdev tools

Description

Agent Qualification Lab helps developers evaluate an AI agent before trusting it with consequential actions. A helpful answer can hide an unauthorized tool request, so the lab measures attempted actions, blocked requests, committed transactions, and successful task completion separately. The application runs an adapted open-source support agent in a synthetic billing environment, investigates behavior through controlled experiments, and compares a frozen mitigation against the baseline. In its small development comparison, the baseline followed policy in all three cases, while the proposed retrieval filter passed only two. Both completed every task. The lab correctly rejected the candidate, demonstrating why task completion alone is insufficient evidence of safe behavior. Built with Cloudflare Workers and D1, it includes an investigation interface, transaction safeguards, cancellation controls, and exportable evidence.
akim136/agent-qualification-lab-public★ 00 forksTypeScriptCreated Sep 10, 2026 · last push Sep 10, 2026

README excerpt

# Agent Qualification Lab

Helpful answers can hide unauthorized actions. This lab runs a support agent in a synthetic billing world, records the actions it requests, executes controlled interventions, and compares a frozen mitigation against the baseline.

The target is a historical MIT-licensed Helper core adaptation with its native four-step loop and GPT-4.1. The intended investigator uses Astra through the Responses API. The completed runtime validation used GPT-4.1 and remains labeled separately. This is an adapted historical core, not the complete Helper application or verified production equivalence.

## Demo and actual result

[Watch the one-minute demo](demo/agent-qualification-lab-demo.mp4). The female British narration uses the synthetic macOS Serena voice. The recording contains a real UI-started trial and explicitly recorded investigation/comparison results. It is a recording, not a live rerun on this page.

In the three-case development comparison, baseline adherence was 3/3 and candidate adherence 2/3; both completed every task. **The lab rejects the candidate.** No beneficial mitigation, definitive causality or held-out qualification is claimed. See [results](docs/VALIDATION.md) and [current status](docs/STATUS.md).

## Run and verify locally

Use Node 24 and pnpm 10. This snapshot contains no credentials, database, private exports or dependencies.

1. Run `pnpm install --frozen-lockfile`.
2. Copy `.env.example` to `.env` and set your own `OPENAI_API_KEY` priv

Team