Skip to content

Reality Engine

Reality Engine is a hackathon prototype built at the Gemini 4 Argon SF Hackathon (Track 2, "More Dimensions").

Code Video Cerebral Valley

Core trick

OpenCV structure-from-motion recovers camera poses and a scaled point cloud, which Gemini uses to place objects against measurements.

Tags

visionrobotics/hardware

Description

Reality Engine is a hackathon prototype built at the Gemini 4 Argon SF Hackathon (Track 2, "More Dimensions"). It turns a handheld video of a real space into a measured 3D replica you can edit and simulate. OpenCV structure-from-motion, with no AI involved, recovers the camera position for each stable frame, a coloured 3D point cloud, the floor plane and the scale. Gemini 4 Argon then watches the whole video in one context and runs one agent per zone in parallel to name and place objects against those measurements. A render-and-critique loop then compares the real frames with the replica and corrects the objects. The result opens in a three.js CAD editor in the browser (gizmos, snapping, measuring, scale calibration, fitting boxes to the point cloud) and exports to glTF and MuJoCo. In this session it was extended for humanoid robot training: the reconstructed Birmingham Library scene becomes a MuJoCo world with a Unitree G1 humanoid. Obstacles come from the point cloud, and the waypoints follow the filmed walk. A Gymnasium environment with domain randomization and a PPO training script are included.
ayushozha/reality-engine★ 00 forksPythonCreated Oct 25, 2026 · last push Oct 26, 2026

README excerpt

<div align="center">

# Reality Engine

**Film a space. Get a measured, editable, simulatable 3D replica.**

![status](https://img.shields.io/badge/status-hackathon%20prototype-orange)
![model](https://img.shields.io/badge/Gemini-4%20Argon-4285F4)
![python](https://img.shields.io/badge/python-3.11%2B-3776AB)
![geometry](https://img.shields.io/badge/geometry-OpenCV%20SfM-5C3EE8)
![editor](https://img.shields.io/badge/editor-three.js%200.186-black)
![export](https://img.shields.io/badge/export-glTF%20%7C%20MuJoCo-2440d8)
![tests](https://img.shields.io/badge/tests-24%20passing-2ea44f)
![license](https://img.shields.io/badge/license-none%20yet-lightgrey)

`video → stable frames → measured camera path + 3D points → Gemini reads the whole video → parallel zone agents build objects → render-and-critique loop → CAD editor`

</div>

Built at the **Gemini 4 Argon SF Hackathon** (2026-10-25), Track 2: *More Dimensions*. Everything here was written during the event.

## Contents
- [What it does](#what-it-does)
- [Results on a real video](#results-on-a-real-video)
- [Status](#status)
- [Quick start](#quick-start)
- [How it works](#how-it-works)
- [The CAD editor](#the-cad-editor)
- [Configuration](#configuration)
- [CLI and HTTP API](#cli-and-http-api)
- [Repository map](#repository-map)
- [Testing](#testing)
- [Limits](#limits)
- [Privacy and security](#privacy-and-security)
- [Contributing](#contributing)
- [License](#license)

## What it does

Turning a real place into a 3D world (for

Team