Skip to content

Robot World

Robot World connects real-world observation with an interactive robotics simulation.

Code Video Cerebral Valley

Core trick

Maps a RealSense depth-camera snapshot of a tabletop into a MuJoCo simulation, using a known-size paper reference for scale.

Tags

robotics/hardwarevision

Description

Robot World connects real-world observation with an interactive robotics simulation. It combines a live Intel RealSense D435 color-and-depth feed, a World Labs virtual environment, and a simulated Unitree G1 humanoid with five-finger Shadow hands. Our latest milestone brings an observed physical tabletop into the virtual world. Using a known-size paper reference, we map objects from a camera capture into simulation, where users can ask the robot to approach and pick up the observed black mug. The interface also supports commands for walking, reaching, and grasping, with MuJoCo simulating object physics and hand contacts. The project explores a practical foundation for bringing real environments into robot development and testing. Live camera viewing and snapshot-based scene mapping work today; continuous reconstruction and object tracking are the next steps.
antopatrex1/robotworld★ 00 forksPythonCreated Sep 8, 2026 · last push Sep 9, 2026

README excerpt

# Robot World — Mac prototype

A local interactive MuJoCo simulation with the **exact [Warm and inviting living room](https://marble.worldlabs.ai/world/7f964e6d-dbdd-4cf7-a8fb-9292001f69ba)** requested by the user, a Unitree G1 with two five-finger Shadow hands, and ten labeled household objects on a physical table.

Open **Start Robot World + RealSense.command** to launch the simulation and camera together. Enter your Mac password in Terminal for the camera helper. It reuses this checkout's Robot World server when already running and restarts the camera helper. Keep both Terminal sessions open; closing the RealSense window ends its camera helper. From a shell, run `./"Start Robot World + RealSense.command"` (add `--demo` for synthetic camera frames).

For the simulation alone, open **Start Robot World.command**, or run:

```sh
.venv/bin/python scripts/patch_viser.py
.venv/bin/python -m robot_world.web_app --open-browser
```

The viewer runs at **http://127.0.0.1:8765**. It stays local to this Mac and needs the Python process to remain running. No World Labs API key is sent to the browser. Local playback and commands do not spend API credits.

## RealSense RGB-D camera

The [realsense](realsense/README.md) folder contains the native RGB-D camera window and MCP capture server imported from `oai6`. Open **realsense/Start RealSense.command**, or run `./realsense/scripts/start.sh`. Use `--demo` for a synthetic preview. Robot World's **RealSense · live camera** panel shares that h

Team