FinalistMost Creative Use of Opus 4.7
rhmoller
A kid raises a hand to the webcam and a puppet comes to life — mouth, eyes, and gestures driven by their fingers via MediaPipe.
Core trick
MediaPipe hand landmarks drive puppet mouth and gestures, while a second model composes unknown props from Three.js primitives and caches them.
Description
A kid raises a hand to the webcam and a puppet comes to life — mouth, eyes, and gestures driven by their fingers via MediaPipe. An AI puppet on stage listens with words AND body language, and replies in character.
When the kid says "let's go to the beach", a sun, a sand castle, and a beach ball appear. When they say "I want sunglasses on my puppet", sunglasses appear. When they say "I want a watermelon hat" — something the catalog doesn't have — the AI says "ooh, let me dream that up!" and a few seconds later a watermelon-shaped hat fades onto its head. Asked again, it pops in instantly from cache.
Behind the scenes, two Claudes work in parallel. A conversation Claude performs the live character with emotion, gaze, and gestures. A separate Claude designs each novel prop on the fly from a tight set of THREE.js primitives. Its compositional reasoning is what makes the co-creation feel real-time.
An interface for play that didn't exist a year ago.
rhmoller/virtual-puppet-theater★ 3212 forksTypeScriptMITCreated Apr 24, 2026 · last push Oct 8, 2026
README excerpt
# Virtual Puppet Theater A virtual puppet theater that runs in your browser. Your webcam tracks your hand with MediaPipe and animates a puppet on stage. Opposite you, a cheerful AI-driven hand-puppet reacts to what you say and do, driven by Claude Opus 4.7. Built for the "Built with Opus" hackathon. > **Note:** The [`hackathon-release`](../../tree/hackathon-release) branch > preserves the exact code submitted to the hackathon. `main` has since > received minor improvements and cleanups made after the deadline. ## Stack - **Bun** — runtime and package manager - **Vite** + **TypeScript** (strict) — frontend build - **three.js** — 3D scene - **@mediapipe/hands** — hand landmark tracking - **Web Speech API** — browser speech recognition; browser synthesis as TTS fallback - **@elevenlabs/elevenlabs-js** — primary TTS, gives the AI puppet an expressive voice - **@anthropic-ai/sdk** — Claude Opus 4.7 drives the AI puppet's dialogue and emotes ## Requirements **Hardware** - A **webcam** — MediaPipe needs it to track your hand. - A **microphone** — speech input drives the conversation. **Browser** - **Chrome** or **Edge** on desktop. The Web Speech API used for speech recognition is best supported there; Firefox and Safari are hit-or-miss. - **HTTPS or `http://localhost`** — `getUserMedia` (camera/mic access) refuses to run on insecure origins. **API keys** - **`ANTHROPIC_API_KEY`** *(required)* — the AI puppet's brain runs on Claude. Get one at [console.anthropic.