Skip to content

XR Wave Agent pipeline to create visionPro + weber creatives

-- Hackathon Summary: Cross-Platform XR Scene Generator What We Built A natural-language-to-3D pipeline that takes English prompts and renders immersive scenes on both visionOS (Apple Vision Pro) and WebXR (any browser)

Code Cerebral Valley

Core trick

A Gemini agent runs observe-reason-act-evaluate loops over an asset directory, emitting one scene.json rendered to USDZ and GLB.

Tags

agentscreative/music

Description

-- Hackathon Summary: Cross-Platform XR Scene Generator What We Built A natural-language-to-3D pipeline that takes English prompts and renders immersive scenes on both visionOS (Apple Vision Pro) and WebXR (any browser) simultaneously, powered by a Gemini autonomous agent. "Place a surfer witch and a punk girl in an ocean scene" → Gemini agent (observe/reason/act/evaluate loop) → scene.json → RealityKit (.usdz) + A-Frame (.glb) → Two platforms, one command Key Innovation: Agent, Not API Call The Gemini integration isn't a single-shot API call — it's an autonomous agent loop: 1. Observe — scans the filesystem for available 3D models and backgrounds 2. Reason — calls Gemini with the prompt + asset context to produce a grounded plan 3. Act — validates assets exist, swaps in fallbacks for missing ones, writes scene.json 4. Evaluate — checks output validity, retries with adjusted prompts (up to 5 times) This means the agent adapts to what's actually available on disk rather than hallucinating assets that don't exist. What It Does - Single command (./run.sh "your prompt") handles environment setup, agent execution, and dual-platform output - Multi-character scenes — place multiple 3D characters with precise positioning - 360° backgrounds — equirectangular panoramas wrapped as immersive sky spheres - Batch mode — generate_all.sh auto-discovers every model in assets/models/ and builds a scene with all of them, no prompt or API key needed - Logical asset mapping — one name ("bruja") resolves to .usdz for RealityKit or .glb for WebXR
romyilano/vercelDeepmind_2026-03-21★ 00 forksPythonMITCreated Mar 21, 2026 · last push Mar 22, 2026

README excerpt

# VercelHackDeepMind

Cross-platform XR pipeline that takes natural language prompts, generates structured scene data via a Gemini-powered autonomous agent, and renders 3D characters and 360° environments in both visionOS (RealityKit) and WebXR (A-Frame).

## Agent Architecture

The Gemini scene agent uses an **observe/reason/act/evaluate** loop — this is what makes it an agent, not just an API call.

```mermaid
graph TD
    Start[Start: User Intent]

    subgraph ObservePhase [1. Environmental Perception]
        Scanner[observe: Scans Filesystem]
        State[World State: Asset Context]
    end

    subgraph ReasonPhase [2. Decision Making]
        LLM[reason: Gemini Brain]
        Plan[JSON Plan: Grounded in Assets]
    end

    subgraph ActPhase [3. Environmental Action]
        Executor[act: Mutates Project]
        Output[scene.json: Final File]
    end

    subgraph LoopPhase [4. Autonomous Correction]
        Evaluate[evaluate: Hardcoded Checks]
        Decision{Valid?}
        Fix[Self-Correction: Alters Prompt]
    end

    Success[Success: Scene Generated]
    Failure[Failure: Max Retries]

    Start --> Scanner
    Scanner --> State
    State --> LLM
    LLM --> Plan
    Plan --> Executor
    Executor --> Output
    Output --> Evaluate

    Evaluate --> Decision
    Decision -- No --> Fix
    Fix -- Retry Loop --> LLM
    Decision -- Yes --> Success
    Fix -- Over 5 Retries --> Failure

    Note1[Narrow Domain] -.-> ObservePhase
    Note2[Strict Pipeline] -.-> R

Team