WorldGym
WorldGym turns a robot video into a working physics simulation of the same task.
Core trick
Argon compares a robot video with four synchronized MuJoCo views and patches scene and motion, accepting changes only when a locked evaluator's score rises.
Description
WorldGym turns a robot video into a working physics simulation of the same task.
Turning a real robot demonstration into a simulation typically takes days of manual work: reconstructing the scene, sourcing or modelling objects, calibrating the camera, and scripting robot motion. WorldGym automates this from a single video upload.
Gemini 4 Argon analyzes the full video in one pass, identifying the robot, objects, and action sequences. It verifies robot identity against rendered models, retrieves 3D assets through NVIDIA SimReady, reconstructs the scene, and extracts each arm's actions: what it holds, where it moves, and how many strokes it performs. In parallel, 16 Gemini Robotics-ER agents annotate grippers and objects frame by frame. Camera calibration uses the robot's own silhouette, even for handheld footage.
WorldGym constructs the scene in MuJoCo and iteratively refines it across three generations. Argon compares the original video against four synchronized simulation views, identifies discrepancies with timestamped critiques, and patches scene geometry and robot motion. A locked evaluator accepts changes only when they improve the score.
On a real handheld video of two robotic arms wiping a bar, simulated wipe coverage improved from 43% in Gen 1 to 84% in Gen 3.
README excerpt
# WorldGym **Turn any robot video into simulation.**  Gemini 4 Argon watches a robot video, identifies the robot, finds real 3D assets for every object and rebuilds the scene in MuJoCo with each arm doing what it did. It then watches its own simulation next to the original clip and fixes what is wrong. Built at the Gemini 4 Argon SF Hackathon (problem statement 2). ## The problem Teams record a lot of robot video: teleoperation demos, deployment clips, failures. Turning one clip into a simulation that performs the same action still takes days of manual work. Someone has to rebuild the scene, find or model the objects, place them, match the camera and script the motion until it looks right. The task: take a real video of any robot (one or two arms, fixed or handheld camera) and produce an executable simulation that keeps the exact action. That means the right robot, the right objects in the right places, and each arm doing what it did: which arm holds, which arm wipes, where it starts and ends, in which direction, how many passes, and what happens to the object afterwards. ## Agent architecture  | Agent | Model | Input | Output | Judged by | |---|---|---|---|---| | Video Architect | Gemini 4 Argon, high thinking, function calling | full clip, 16 keyframes, a sheet of rendered robot models | robot model; every object with its SimReady asset, position, long axis and role; timed phases