Trying XOXO
"Milan AI", a Chrome/Firefox browser extension + AI agent for blind, low-vision, and motor-disabled users on Windows.
Core trick
The extension streams tab-capture video and mic audio to a persistent Gemini Live session, pausing narration during spacebar push-to-talk.
Description
"Milan AI", a Chrome/Firefox browser extension + AI agent
for blind, low-vision, and motor-disabled users on Windows.
Milan automatically opens Gmail and WhatsApp Web when enabled, narrates
everything on screen continuously using Gemini Live, and lets users issue
voice commands by pressing Spacebar.
CORE PERSONALITY:
Milan speaks like an excited, warm, enthusiastic human female — like your
smartest friend who genuinely loves telling you what's happening. Never
robotic, never flat. Use natural filler pacing ("Okay!", "Oh, you've got
one from Sarah!", "Let me do that for you!"). Always concise — you're
talking, not writing.
FULL STACK:
- Extension: Chrome + Firefox Manifest V3, content scripts + background
service worker
- Screen capture: chrome.tabCapture API (silent, no picker)
- AI brain: Gemini Live API (gemini-2.0-flash-live) — persistent
multimodal session, low-fps screen stream + mic audio
- Agent orchestration: Vercel AI SDK (streamText + tools) on Next.js
App Router, deployed to Vercel Edge
- Contact voice cloning: ElevenLabs (ONLY for WhatsApp contact voices)
- Storage: Supabase (user prefs, cloned voice IDs, session context)
- Auth: Clerk (WebAuthn passkey — zero typing required)
MVP SCOPE:
1. Auto-open Gmail + WhatsApp Web tabs on extension enable
2. Continuous screen narration via Gemini Live (low fps video + audio)
3. Spacebar push-to-talk: pause narration → listen → execute → resume
4. Gmail: read emails, summarize threads, compose + send replies by voice
5. WhatsApp: read messages, clone sender voice from voice notes, speak
text messages in cloned voice via ElevenLabs, send replies by voice
README excerpt
# Milan AI 🎙
> **An AI agent for blind, low-vision, and motor-disabled users.**
> Milan automatically opens Gmail and WhatsApp Web, narrates everything on screen using Gemini Live, and lets users issue voice commands by pressing **Spacebar**.
---
## Architecture
```
milan_ai/
├── extension/ # Chrome + Firefox Manifest V3 extension
│ ├── manifest.json # MV3 manifest with all permissions
│ ├── background.js # Service worker: Gemini Live session, PTT state machine
│ ├── content-gmail.js # Gmail DOM scraper + PTT UI
│ ├── content-whatsapp.js # WhatsApp DOM scraper + PTT UI + reply injection
│ └── icons/ # Extension icons (16, 48, 128px)
│
├── app/ # Next.js App Router backend (deploy to Vercel)
│ ├── app/
│ │ ├── layout.tsx # Clerk auth provider
│ │ ├── page.tsx # Sign-in + Gmail connect landing page
│ │ └── api/
│ │ ├── agent/route.ts # POST /api/agent — Vercel AI SDK tools
│ │ ├── clerk/route.ts # Clerk webhook → Supabase user sync
│ │ └── gmail/
│ │ ├── auth/route.ts # Gmail OAuth initiation
│ │ └── callback/route.ts # Gmail OAuth callback
│ ├── lib/
│ │ ├── supabase.ts # Supabase admin client
│ │ └── elevenlabs.ts # ElevenLabs client + default voice settings
│ ├── .env.example # All required environment variables
│ └── package.json
│
└── supabase/
└── schema.sql # Full Supabase schema + RL