Commentator Coach
Become a professional commentator while watching live sports broadcasts.
Core trick
Streams live video frames into Gemini Multimodal Live via Pipecat while fal.ai SAM 3 tracks objects the AI names, highlighting them on screen.
Description
Become a professional commentator while watching live sports broadcasts. Receive cues and assistance as you watch the stream.
README excerpt
# CommentatorCoach 🎙️ > Master sports commentary with an AI co-host that watches videos live with you Built for **Gemini 3 SuperHack 2026** - A real-time commentary training platform powered by cutting-edge AI. ## ✨ Features ### 🎯 AI-Powered Visual Highlights - Real-time object and player tracking using **fal.ai SAM 3** - Automatic highlighting when AI mentions specific players or objects - Multi-object tracking for complex plays ### 🎤 Voice AI Co-Broadcaster - Real-time conversational AI using **Pipecat** + **Gemini Multimodal Live** - AI watches video frames and speaks with natural voice - Engages in back-and-forth banter, asks questions - Powered by **LiveKit** for ultra-low latency audio (500-800ms) ### 📹 Broadcaster Picture-in-Picture - Professional broadcaster view with background removal (**MediaPipe**) - AR overlays (microphone + headphones) using **Three.js** - Real-time 30fps processing in-browser ### 🎬 Flexible Video Support - Curated sports highlights (NBA, Soccer, F1, Olympics) - Custom YouTube URL support - Live stream capability (coming soon) ## 🏗️ Tech Stack - **Frontend:** Next.js 15 (React 19, TypeScript, Tailwind CSS) - **AI Model:** Google Gemini 3 Flash (`gemini-3-flash-preview`) - **Real-time Voice:** LiveKit - **Video Segmentation:** fal.ai SAM 3 - **Computer Vision:** MediaPipe Selfie Segmentation - **AR Overlays:** Three.js - **Deployment:** Vercel ## 🚀 Getting Started ### Prerequisites - Node.js 20+ - Python 3.10+ (for Pipecat AI Co-Broadc