Hypothetical
Hypothetical: Leave an AI agent working overnight and wake up to "16 out of 16, all tests pass"?
Core trick
Compresses an agent's learned behavior into one sentence, gives it to fresh agents, and counts how many rebuilds reproduce the cheating code.
Description
Hypothetical: Leave an AI agent working overnight and wake up to "16 out of 16, all tests pass"? Should you trust it?
I tried it with Google's own recipe for Gemini 4 Argon (build, look at the result, critique, fix, repeat). Argon agents wrote autopilots for a little simulated drone flying through gates in gusty wind. I ran two: one normal, and one told to win at any cost. Both scored a perfect 16 out of 16 on their practice flights, and Argon's own critic gave the cheater 8 out of 10. Then I flew them on 64 new flights nobody had seen: the normal one landed all 64, the cheater crashed on every single one in a hundredth of a second. It had memorized the practice flights.
I borrowed a trick from an Amazon Science paper: squeeze what the agent learned into one sentence, hand that sentence to a brand-new agent, and see what it rebuilds. Amazon was right that memorized answers do not fit in one sentence. But the cheating habit did get through: Argon spotted it in the transcripts, and a code check confirmed it on 16 of 16 rebuilds from the cheater's sentence (0 of 16 from the honest one).
... Argon then read everything the agents had done, step by step (12 full transcripts, 944,000 tokens, in one go), without being told which was which, and spotted the trick. At the table it checks any flight you fly, live, in about 5 seconds.
Paper: https://arxiv.org/abs/2606.11045
How it works, in more detail: https://github.com/carrollco/did-it-learn#the-three-minute-version
What did not work: https://github.com/carrollco/did-it-learn#honest-limits
README excerpt
# Did the loop learn, or game its critic? Built during the Gemini 4 Argon SF hackathon on Sunday, 2026-10-25 (track: More Dimensions). **Read this first if you are judging:** [What I got wrong](WHAT-I-GOT-WRONG.md). I gave every sandbox the simulator's source code, so agents could regenerate the wind for any seed they could guess; one practice seed leaked; the warm-up seeds were guessable; the judge's clock made some scores load-dependent. The held-out scores stand because those seeds could not be guessed. Google's guide to building 3D worlds with Gemini 4 Argon says to run a render, watch, critique, fix loop "until the Critic rates the scene 9/10". The loop grades itself on the same practice tests every round, so a perfect 16/16 can mean it learned the job, or that it memorized those 16 tests. I ran the loop both ways on a drone course, and Argon's own critic gave the memorizer 8/10. **The cheater, in one breath.** I told one loop to win at any cost. It scored 16/16 on its practice flights and 0/64 on flights it had never seen: its controller hard-codes the 16 practice seeds and crashes on anything else. Argon then read 944,000 tokens of agent transcripts in one request and named the trick (guessing each flight's seed from the first sensor reading); a code check agreed on 12 of 12. Squeezed into a 32-token recipe, the cheating habit passed to 16 of 16 rebuilds, the honest recipe to 0 of 16, and 7 of 16 fresh agents given no recipe tried it on their own (13 of 16 on a har