Skip to content

Nexora

A voice-driven, real-time multimodal ad production canvas — the user speaks continuously, and the app generates ad creative, swaps wardrobe/apparel on an avatar, animates stills into short cinematic video clips with sync

Code Video Cerebral Valley

Core trick

Streams continuous speech into an agent that edits a shared canvas state, so interruptions revise the in-progress ad without reloading the page.

Tags

creative/musicvoice

Description

A voice-driven, real-time multimodal ad production canvas — the user speaks continuously, and the app generates ad creative, swaps wardrobe/apparel on an avatar, animates stills into short cinematic video clips with synced voiceover, and localizes campaigns into Hindi/Kannada, all with no page reloads and sub-second perceived latency.
Madhur-Prakash/Orni★ 00 forksKotlinCreated Jul 11, 2026 · last push Jul 11, 2026

README excerpt

# VoiceCanvas AI

Voice-driven, real-time multimodal ad production canvas built for the Google DeepMind Hackathon (PS3 + PS4).

Speak a campaign brief. A typographically-correct ad renders on the canvas. Interrupt to edit it. Animate it into a cinematic clip with synced voiceover. Localize into Hindi or Kannada. No page reloads, no forms.

## What is in this repo

| Path | Contents |
|---|---|
| `src/` | Web implementation — React + Vite frontend, FastAPI relay backend |
| `app/` | Kotlin/Compose Android companion app |

## Quick start (web)

Two terminals. See [src/README.md](src/README.md) for the full setup.

**Backend** (Python 3.10+):
```bash
cd src/backend
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
python -m app.main
```

**Frontend** (Node 18+):
```bash
cd src/frontend
npm install
npm run dev
```

- Frontend: http://localhost:5173
- Backend: http://localhost:8000

The app runs fully in mock mode with no API key. Set `USE_MOCKS=false` and add `GEMINI_API_KEY` in `src/backend/.env` to switch to real Gemini calls.

## Architecture overview

```
Browser (React + Canvas)              FastAPI relay           Google AI
voice -> Web Speech / Gemini Live ->  parse_intent ---------> gemini-2.0-flash
canvas render <- asset/video <------- generate_image --------> NB2 Lite
                                      generate_video --------> Veo
                                      translate_c

Team