Skip to content

Keller

Keller — a voice-first agentic browser for low-literacy Indians.

Code Video Cerebral Valley

Core trick

Replaces Aadhaar and PAN values with placeholders before any cloud prompt, then fills real values locally after spoken confirmation.

Tags

accessibilityvoicesecurity

Description

Keller — a voice-first agentic browser for low-literacy Indians. Speak a task in Hindi ("fill my pension form on the Jan Seva portal") and an on-device AI agent drives a real web browser for you — reading the page, clicking, and filling forms — while narrating each step back in simple Hindi. A built-in privacy firewall keeps sensitive IDs (Aadhaar, PAN, mobile) on the phone: the cloud AI only ever sees placeholders like [AADHAAR_1], and the real values are filled locally after a spoken confirmation. Documents like ration cards are read by on-device OCR (ML Kit Devanagari), never leaving the device. Built as a minimal Android WebView shell (com.keller.browser), powered by Gemini (Live API for voice, function-calling for the browser agent), with a local offline mock government portal so it works even with no venue network.
aabid2947/keller★ 00 forksKotlinCreated Jul 11, 2026 · last push Jul 11, 2026

README excerpt

# Keller — the browser that talks (and keeps your secrets)

Voice-first agentic Android browser for the 250M+ Indians the text web left behind.
Speak in Hindi; a multi-agent system (Gemini Live API voice agent + a Gemini
function-calling Executor agent) sees the page, plans, and acts — clicks, fills,
scrolls — while a PII Guard agent with VETO power guarantees your Aadhaar/PAN are
filled locally on-device and NEVER appear in any cloud prompt (placeholders only).
Camera + on-device ML Kit OCR reads physical documents (ration card) straight into
web forms.

**Built entirely on 2026-07-11 at the Google DeepMind Bangalore Hackathon**
(Problem Statement 2: Autonomous Orchestration with Managed Agents).

Named for Helen Keller.

## Build (no Android Studio needed)
- JDK 17, Android SDK at `local.properties` sdk.dir, Gradle 8.14
- `gradle assembleDebug` then `adb install -r app/build/outputs/apk/debug/app-debug.apk`
- Put `GEMINI_API_KEY=...` in `.env` at repo root (never committed)

## Drive it without speaking (debug channel)
`adb shell "am broadcast -n com.keller.browser/.debug.DebugCommandReceiver --es cmd 'your command'"`
Free text = agent goal. Verbs: `portal · open <url> · map · overlay on/off ·
click <id> · fill <id> <v> · read · scroll · shot · voice on/off · say <text> ·
ocr · camera · cancel` (= voice barge-in).

## Architecture
Voice (Gemini Live, raw WSS, sync-only tool calls) -> TaskBus -> Executor agent
(Gemini function-calling loop over PII-redacted DOM maps; PII Gua

Team