Finalist
StateShift
StateShift efficiently finds semantic events in long videos using Gemini and classic search algorithms.
Core trick
Compares Gemini classifications at video endpoints, then runs exponential then binary search over sampled frames to locate the state transition.
Tags
Description
StateShift efficiently finds semantic events in long videos using Gemini and classic search algorithms. A user uploads a video and asks a natural-language query like “Find when the bottle gets knocked over” or “Find when the door opens.” Instead of scanning every frame, the system first compares the semantic state at the start and end of the video. If they differ, it performs a binary search to localize the transition. If not, it uses exponential search (1, 2, 4, 8, …) to discover a likely bracket before refining with binary search. In the common single-transition case, this reduces Gemini calls from linear to near-logarithmic growth, often cutting model usage by 99%+ on long videos. This reduction makes using Gemini for these sorts of tasks a complete no-brainer, rather than a potentially expensive solution. It also expands the possible videos that can be analyzed beyond what can be handled by simple CV models. Two demo videos are included; see the README for setup and usage.
README excerpt
# StateShift StateShift is an MVP for localizing semantic state changes in long videos with Gemini and a search strategy that uses far fewer model calls than exhaustive frame-by-frame scanning. Given a video and a query like "Find when the bike disappears" or "Find when the door opens", the app: 1. uploads the video 2. extracts sampled frames with `ffmpeg` 3. asks Gemini to classify only selected sampled frames 4. uses bracketing + binary search to estimate the transition timestamp 5. shows before/after evidence frames, confidence, explanation, and the search trace ## Quick Start Everything application-specific is in this repo, but to run it locally you still need: - Node.js 20+ - npm 10+ - `ffmpeg` installed and available on `PATH` - a valid Gemini API key Setup: ```bash npm install cp .env.example .env.local ``` Then add your `GEMINI_API_KEY` to `.env.local`. Run locally: ```bash npm run dev ``` Build for production: ```bash npm run build ``` Open [http://localhost:3000](http://localhost:3000). ## How To Demo 1. Start the app with `npm run dev` 2. Upload a local video with a state change (although the state can also change back) 3. Enter a query that matches the action in the video, such as: - `Find when the bike disappears` - `Find when the package is removed` - `Find when the door opens` - `Find when the room becomes empty` 4. Keep the default sampling rate of `1` fps for the initial demo 5. Run analysis and inspect the returned timestamp, evide