Aunty AI
Aunty AI is a real time voice coach for bargaining in India.
Core trick
Diarizes speakers from a 3-second enrollment voice embedding, then whispers a reason plus scripted line into the earbud within about a second of the seller pausing.
Description
Aunty AI is a real time voice coach for bargaining in India. You wear earbuds, you talk to the seller in Hindi or Hinglish, and Aunty whispers a coaching line in your ear while you're talking to him. Each whisper has two parts: a quick reason (the angle, why this is the right move) and the literal line you should say next. So instead of just "say 380", you hear "wo upar se kheench raha hai, neeche se anchor laga do, bolo: 380 mein de do."I built this because I bargain badly. Most people in India do. We grow up doing it but nobody teaches you the floor price, the timing, when to walk, when to play the regular-customer card. The bhaiya across the counter does this 200 times a day, you do it twice a year. The result is everyone overpays.The hard part is latency. The whisper has to land in your ear within about a second of the seller pausing or the rhythm of the haggle breaks. To get that I'm running gpt-realtime over a LiveKit WebRTC connection from a native iOS app, with Sarvam handling the Indian-language ASR side. There's also a small speaker diarization piece on top: at the start of the session you record three seconds of yourself talking, the server runs that through a WeSpeaker ONNX model and gets a voice embedding, and after that every utterance gets labeled USER or SELLER deterministically before Aunty replies. Without that the model gets confused about whose turn it just was. The persona prompt forces the angle plus line shape on every response, plus a small safety regex that blocks Aunty from telling you to lie, threaten or insult.Built end to end in a few sessions with Claude Code: backend in Go, app in Swift, prompts in markdown.
README excerpt
# Aunty AI Real-time voice negotiation "whisperer" for India (Hindi / English / Hinglish). The user speaks with a seller. Earbuds feed private one-line coaching. The product lives or dies on **time-to-first-whisper-audio of 1.0–1.5s** — latency is the feature. > **Architecture note (2026-05-23 re-architecture).** The transport is a **WebSocket bridge** (LiveKit was removed entirely) and each session runs a **single realtime runtime** — **Gemini Live is the production default**, with OpenAI Realtime and xAI Voice Agent as alternative runtimes. The old cascade STT→LLM→TTS pipeline was removed. See [`ARCHITECTURE.md`](ARCHITECTURE.md) and `docs/superpowers/specs/2026-05-23-aunty-rearch-websocket-gemini-design.md` for the rationale, and [`CLAUDE.md`](CLAUDE.md) for conventions. ## Repo layout ``` aunty-ai/ ├── services/ │ ├── session-actor/ Go — per-session actor, realtime-runtime strategy, provider adapters │ │ ├── providers/ RealtimeRuntime adapters (Gemini/OpenAI/xAI) + external TTS + WeSpeaker ONNX encoder │ │ ├── strategy/ e2e.go — speaker labels, ManualResponse, [CONTEXT], persona_kb + state grounding │ │ ├── transport/ ws_bridge.go — bidirectional WebSocket, base64 PCM16 envelopes │ │ ├── audio/ noisesuppression/ — vendored rnnoise CGO pre-processor (16 kHz Int16 PCM) │ │ ├── safety/ regex fast validator + output_gate.go (200 ms hold) │ │ ├── auth/ session bearer-JWT verification (shared secret with we