Skip to content
Finalist

slava m

next-state turns real-world videos into live simulations.

Code Video Cerebral Valley

Core trick

Compiles a short indoor video through Gemini into a 3D scene graph, then runs utility-scored agents over its navigation graph.

Tags

visionagents

Description

next-state turns real-world videos into live simulations. when a person enters a room, they do not just see who is there — they instinctively infer who people are and what might happen next. ai is getting good at scene understanding, but it still struggles to anticipate near-term real-world dynamics. we call that missing layer situational intuition. my project is an attempt to give ai that intuition.
slaviquee/next-state★ 10 forksTypeScriptCreated Mar 14, 2026 · last push Mar 14, 2026

README excerpt

# next-state

Video-to-3D scene compiler with live AI-driven agent simulation. Upload a short video of any indoor space and watch it come alive as an interactive 3D scene populated with autonomous agents.

## How it works

```
Video Upload → Gemini Analysis → ADK Compile Pipeline → CompiledScenePackage JSON → Three.js Simulation
```

1. **Upload** a short video (up to 20s) of an indoor space
2. **Gemini 3.1 Pro** analyzes the video — detects objects, people, zones, spatial layout
3. **ADK pipeline** compiles the analysis into a structured 3D scene with agents, furniture, navigation graphs
4. **Three.js frontend** renders the scene and runs a live simulation with utility-based AI agents

The server is stateless at runtime — the browser owns all live world state.

## Tech stack

| Layer | Choice |
|-------|--------|
| Frontend | Vite + React 19 + Three.js r183 + R3F v9.5 |
| State | Zustand |
| 3D Renderer | WebGPURenderer (auto-fallback to WebGL 2) |
| UI | shadcn/ui + Tailwind CSS |
| Backend | Node.js + TypeScript |
| AI | Gemini 3.1 Pro/Flash-Lite via `@google/genai` |
| Orchestration | `@google/adk` v0.4.0 |
| Validation | Zod |
| Testing | Vitest |

## Getting started

### Prerequisites

- Node.js 20+
- A [Google AI](https://aistudio.google.com/) API key (paid account)

### Setup

```bash
# Clone and install
git clone <repo-url> && cd next-state
npm install

# Configure environment
cp .env.example .env  # or create .env manually
```

Add your API key to `.env`:

```env
GE

Team