GTA San Astra
GTA San Astra gives GPT-6 Astra a controller and asks it to drive in Grand Theft Auto: San Andreas using screenshots as its only sensor.
Core trick
Screenshot-only loop sends frames to the model, which picks controller inputs and compares successive frames to judge results.
Description
GTA San Astra gives GPT-6 Astra a controller and asks it to drive in Grand Theft Auto: San Andreas using screenshots as its only sensor. It is a visual autonomy experiment: can a general-purpose model learn steering, braking, lane positioning and recovery by observing the consequences of its own actions?
A native macOS bridge connects a PS2 emulator to a Python control loop. Astra selects every driving action, compares successive screenshots, and carries forward concise route and vehicle-dynamics notes. We tested both paused thinking and a mode that keeps the world moving at half speed during inference. Native recordings, frame exports and decision logs make each attempt inspectable.
The prototype can navigate roads and complete individual turns, but still makes lane, collision and recovery errors. A full return around the block is not yet proven. The project turns a familiar game into a concrete test of visual reasoning and closed-loop control.
README excerpt
# GTA San Astra ## Full development session: learning to drive Watch the end-to-end development session for GTA San Astra: building and iterating on the experiment to teach GPT-6 Astra to drive in *GTA: San Andreas*. https://github.com/user-attachments/assets/bd12540a-aba6-4bd0-ba82-dd2d999ba898 This 55-second desktop timelapse includes all 332 saved screenshots from September 8, 2026, 10:25 a.m. to 4:38 p.m. Pacific. Screenshots were taken every five minutes initially, then every minute, and play back at six per second. It captures the development process across the session; driving outcomes are documented separately below. [Video and session details](https://github.com/PandelisZ/gta-san-astra/issues/2) **Give GPT-6 Astra a controller, a screenshot, and the next few frames.** GTA San Astra is a visual driving experiment inside the PlayStation 2 version of *Grand Theft Auto: San Andreas*. Astra looks at the game, chooses controls, and sees what happens next. A native macOS bridge turns those decisions into PCSX2 input. The current demo runs actions at normal speed and keeps the world moving at 50% speed while Astra decides. Earlier experiments paused during inference. Astra builds the experiment and becomes its driving policy. The question is concrete: **can a visual model follow a road when pixels are its only sensor?** [Two-minute demo guide](docs/DEMO.md) · [Recorded evidence](docs/EVIDENCE.md) · [Live validation](docs/TESTING.md) · [Repository](https://github.com