Skip to content

Zero To World

Zero to World: upload a scan of any room, train a robot in that space.

Code Cerebral Valley

Core trick

Projects Gemini-detected room object bounds into 3D coordinates and compiles them into a MuJoCo scene for a G1 locomotion policy.

Tags

robotics/hardwarevision

Description

Zero to World: upload a scan of any room, train a robot in that space. We built a pipeline that turns a Gaussian Splat (.ply) of any real room into a working robot training simulation. Upload your scan, and within minutes Gemini 3 Flash analyses the space, detecting every table, chair, wall and obstacle via multimodal vision. Those bounds are projected into 3D coordinates and compiled into a MuJoCo physics scene that perfectly mirrors your real room. A Unitree G1 is then deployed into that simulation, using a pre-trained locomotion policy to navigate the actual layout around the real furniture, toward the real destination. The long-term vision: every room on Earth becomes a robot training ground. No 3D artists. No hand-crafted simulators. Just scan, upload, deploy.
afifi-yusuf/ZeroToWorld★ 00 forksSwiftCreated Mar 21, 2026 · last push Mar 25, 2026

README excerpt

# Zero to World

Web app that turns a 3D Gaussian splat of a space into scene understanding and a MuJoCo-style simulation: view a splat, capture viewport frames, analyze them with Google Gemini, and generate MJCF for robot training overlays.

The UI is built with **Next.js 16**, **React 19**, **Three.js** / **React Three Fiber**, and **Gaussian Splats 3D** for `.ply` viewing.

## Repository layout

- **`zero-to-world/`** — Next.js application (this is where you install dependencies and run the dev server).

## Prerequisites

- **Node.js** 20+ (matches `@types/node` in the project).
- **Google AI (Gemini) API key** for scene analysis (`/api/analyze-splat`).

Optional, depending on how you deploy or integrate hardware:

- **Vercel Blob** — session JSON and uploaded `.ply` files use `@vercel/blob`. On Vercel, the token is usually injected automatically; locally you need a blob token (see [Vercel Blob environment variables](https://vercel.com/docs/storage/vercel-blob/using-blob-sdk#environment-variables)).
- **WebSocket services** — relay or MuJoCo bridges if you use those features (see env vars below).

## Getting started

```bash
cd zero-to-world
npm install
# Create .env.local with the variables in the table below (at minimum GEMINI_API_KEY for analysis).
npm run dev
```

Open [http://localhost:3000](http://localhost:3000).

Other scripts:

| Command        | Description        |
| -------------- | ------------------ |
| `npm run dev`  | Development server |
| `npm run build`

Team