wevr

🎬 Inspiration Every memorable interactive story relies on a complete production crew: a writer crafting dialogue, a set designer establishing the scene, a composer scoring the mood, and a director making sure choices made in Act I aren't forgotten by Act III. For solo creators, tabletop storytellers, and indie writers, that crew simply doesn't exist. Traditional game engines demand heavy coding and complex tooling, while generic AI chatbots sit at the other extreme: amnesiac text windows with no durable memory, no persistent state, and no real rules. We built wevr to turn that solo experience into a full AI-native production studio. Our goal was to give any writer an autonomous production crew powered by agentic AI: Studio: Creators design living worlds, factions, characters, deterministic win/lose conditions, generated scene art, and an original score composed with Google Lyria, without writing a single line of code. Play: Audiences experience that world turn by turn. Google Gemini takes the director's chair, streaming prose, generating per-turn scene illustrations, adjusting the musical mood live, and honoring the author's world rules through validated tool calling. 🛠️ How We Built It We built wevr as a decoupled, multi-service architecture designed to separate standard application state from heavy AI orchestration and streaming. 1. System Architecture Frontend ( apps/frontend ) Built with React 18 , Vite , TypeScript , Tailwind CSS , Zustand , and TanStack React Query . It delivers two main surfaces: Studio A no-code authoring workspace for: Entities Factions Facts Conditions End conditions Cover art and music generation It also features AERO , an AI co-designer sidebar that inspects scenario state and proposes atomic, reviewable changes such as: Entities Conditions Plot hooks Creators can apply suggestions with a single click. Play An atmospheric turn-by-turn console featuring: Real-time Server-Sent Events (SSE) prose streaming Inline scene illustrations Dice rolls Dynamic soundtrack crossfading Spectator support Core API ( apps/core-api ) Built with Python , FastAPI , async SQLAlchemy , and PostgreSQL . Responsible for: Authentication Scenario CRUD Published game discovery Reviews Bookmarks Background jobs for scenario asset generation Turn Resolution Service ( apps/turn-resolution-service ) A dedicated async Python/FastAPI service responsible solely for turn orchestration. For every turn it: Loads the player's current state Retrieves relevant memory context Prompts Gemini Validates tool calls Triggers scene image generation Streams narrative output via SSE 2. Google Cloud & Partner Integrations Google GenAI SDK ( gemini-3.5-flash-lite ) Powers the live story narrator in the Turn Resolution Service by: Streaming prose token-by-token Driving gameplay logic through structured tool calls Google GenAI SDK ( gemini-3.1-flash-image ) Automatically generates a fresh visual scene illustration for every turn and renders it directly inside the player's narrative feed. Vertex AI (Google Lyria lyria-002 ) Generates original thematic musical scores for authored scenarios directly inside Studio. Replit (Partner Integration) Creators can build custom mini-games using our Replit starter kit template. At runtime, wevr embeds the Replit game in a sandboxed iframe and executes a two-way handshake over the browser's postMessage API: ready() reportResult() This allows minigame outcomes to directly update game state. 3. Persistent Memory Layer (HydraDB) To eliminate LLM amnesia across long playthroughs, we developed a dedicated long-term memory layer combining: PostgreSQL pgvector Graph storage Bitemporal Fact Tracking Tracks both: Valid Time : When an event occurred in the narrative Transaction Time : When the system recorded the event This prevents narrative contradictions and enables consistent world state. Template Memory Cloning Scenarios are ingested into a base memory template once during authoring. When a player starts a new playthrough: The template space is cloned instantly Individual playthrough memories remain isolated Scaling remains efficient ⚡ Challenges We Faced 1. Eliminating State Hallucinations (Validate-Before-Apply) LLMs prompted to simply "track health and inventory" inevitably hallucinate. They create non-existent items, ignore statutory limits, or overlook critical quest requirements. How We Solved It We established a strict Validate-Before-Apply boundary (ADR-4). Gemini cannot mutate game state directly in prose. Instead, it proposes structured tool calls such as: update_entity_stat inflict_condition trigger_ending The Turn Resolution Service validates these calls against author-defined Pydantic schemas and rules before writing changes to PostgreSQL. If a tool call violates scenario rules, it is rejected safely without breaking the narrative. 2. Turn Latency and Multi-Model Coordination A single player turn involves: Memory retrieval LLM reasoning State updates Scene image generation Audio mood cues Running these sequentially created unacceptable delays. How We Solved It We split the pipeline into prioritized synchronous and asynchronous paths. Narrative begins streaming immediately via gemini-3.5-flash-lite Scene image generation runs concurrently via gemini-3.1-flash-image Generated images appear as soon as rendering completes Memory indexing writes are batched rather than blocking active gameplay 3. Graceful Multimodal Degradation Cutting-edge models and cloud quotas occasionally fail or encounter rate limits. If music or image generation fails, gameplay should continue uninterrupted. How We Solved It We built safe fallback boundaries into generation clients. Image generation failures return None Narrative completion proceeds normally Audio generation failures automatically fall back to curated ambient stems The player's experience remains intact even when auxiliary services fail. 4. Dynamic Audio Crossfading in the Browser Static loops quickly become repetitive, while abrupt transitions break immersion. How We Solved It Gemini tags narrative chunks with ambient mood cues such as: [MOOD: calm] [MOOD: melancholy] The frontend Web Audio manager listens for these SSE events and performs smooth gain-curve crossfades between mood-matched tracks. This keeps the soundtrack dynamically aligned with the story. 💡 What We Learned 1. Immersion Requires Multimodal Synergy, Not Just More Parameters Fast, responsive text combined with contextual scene art and a synchronized soundtrack creates a far more convincing cinematic experience than a larger model generating text alone. 2. AI Works Best as a Co-Designer, Not an Autocomplete Engine When building authoring tools like AERO, creators didn't want an AI that hijacked their story. They wanted an assistant that: Inspects the current world Identifies gaps Proposes concrete, reviewable cards Creators remain in control while benefiting from AI-assisted design. 3. Hard System Boundaries Enable Creative Freedom Allowing an LLM to improvise text freely while restricting its authority over game state through strict validation provides the best of both worlds: Uninhibited roleplay and storytelling Real, enforceable stakes 🚀 What's Next for wevr Speech-to-Speech Character Dialogue Add real-time voice input and character-specific dialogue generation so players can converse naturally with NPCs. Playthrough Branching ("Forking") Allow spectators to jump into any shared playthrough at critical decision points and explore alternate timelines. Expanded Creator Ecosystem Support: Multi-act scenario campaigns Collaborative co-authoring A public community library for sharing custom Replit minigames