HackOverflow
What if we give Gemini Live model the ability to annotate on screen?
Core trick
Gemini Live watches a live camera feed, draws on-screen annotations to ask clarifying questions, and turns the annotated session into a guide.
Description
What if we give Gemini Live model the ability to annotate on screen? That's the idea we started with. The problem we thought of solving was every generation, there are so many traditional skills in India that die because they are undocumented on the internet. Through Heir, you can document all those traditional skills. In the local language, the craftsman can speak, and then you can create a guide in English for future generations and for AI training. The product can annotate on screen, and a human can also annotate on screen. It'll try to fully understand what the human says about the skill and then create a guide with images annotated and full length text guide at the end.
README excerpt
# Heir **The AI is the apprentice. The human is the master.** Built at the Google DeepMind Bangalore Hackathon (July 2026). India's rarest skills were never written down — they exist only as hand-motion, in the muscle memory of aging masters. Rogan art survives in a single family in Nirona, Kutch. When knowledge lives only in a curve drawn in the air, no book, video, or chatbot can capture it. So we inverted every "AI teaches humans" demo: **Heir watches a master work through a live camera, converses by voice (in the master's language), and learns the skill — proactively.** It asks *"what is this?"* while drawing an arrow on the thing it means. It confirms understanding by drawing what it understood — the stroke path, the fold axis — back onto a photo of the master's own work. The master corrects it by drawing directly on the AI's drawing with a finger (**bidirectional ink**). It files cropped snapshots with observations into its field notebook. And when the session ends, one button compiles everything into an **illustrated, step-annotated curriculum** — with a second model (the "study brain") reviewing the session against a mastery checklist and negotiating with the live apprentice until the skill is fully documented. Any skill. Any master. Documented before it's gone. ## How it works ``` Browser (Vite + React + TS) FastAPI backend (thin) ├── WS direct ↔ Gemini Live ├── POST /api/session → Live WS auth │ (gemini-3.1-flash