Aawaaz
Aawaaz (आवाज़) Real-Time Voice Agent for Rural Welfare Access 430 million Indians are functionally illiterate.
Core trick
Gemini Live handles Hindi voice, and sensitive PII switches the session to on-device Gemma 4 with Tesseract OCR and encrypted SQLite.
Description
Aawaaz (आवाज़) Real-Time Voice Agent for Rural Welfare Access
430 million Indians are functionally illiterate. They qualify for government welfare schemes PM-KISAN, PMAY, Ayushman Bharat, Ujjwala but can't navigate the portals to claim them. Aawaaz fixes that through real-time multimodal voice interaction not a chatbot with a microphone stapled on.
Aawaaz uses the Gemini Live API for continuous, interruption-aware Hindi conversation. The agent listens in real time, extracts profile fields mid-speech without waiting for turn completion, and responds with natural Hindi adjusting tone when the user sounds confused or hesitant. It reads vocal cues, not just words.
Aawaaz has a hard privacy boundary baked into the agent loop. The moment a user speaks or scans sensitive PII Aadhaar number, bank account, IFSC code the agent automatically switches from Gemini cloud to Gemma 4 E2B running fully on-device. That data is processed locally, stored encrypted in SQLite, and never transmitted to any server. When the sensitive exchange is complete, the agent seamlessly returns to Gemini Live for the rest of the conversation. The user never has to think about this the agent handles the boundary autonomously.
Local Tesseract OCR scans Aadhaar and ration cards entirely on-device. Once the profile is complete, Aawaaz opens the actual government portal pmkisan.gov.in, pmayg.nic.in and fills every visible form field autonomously via Playwright, stopping only at OTP the one action only the user can take.
Tech stack: Gemini Live API · Gemini Flash · Gemma 4 E2B (PII + offline) · FastAPI · Playwright · Tesseract OCR · SQLite (encrypted) · Web Speech API
README excerpt
# 🏛️ आवाज़ — AAWAAZ
### **Enterprise Edge-to-Cloud Voice & Form Automation Infrastructure for India's Digital Public Infrastructure (`DPI`)**
*Built for the Google DeepMind Bengaluru Hackathon · Powered by Gemini 2.0 Flash, Local Gemma 4 Engine, and Intelligent Live DOM Automation*
---
## 🌟 Executive Summary & Architecture Vision
In India, over **800 million rural citizens** are eligible for life-changing central and state welfare schemes (`PM-KISAN, PM Fasal Bima Yojana, e-Shram, MGNREGA, Ayushman Bharat`). Yet, **less than 15%** successfully enroll due to three structural barriers:
1. **Linguistic & Digital Literacy Walls**: Official portals (`pmkisan.gov.in`, `pmfby.gov.in`) are complex English web applications requiring multi-step ASP.NET form navigation.
2. **Connectivity Deserts**: Rural habitations frequently suffer from 2G drops, packet loss, or total air-gapped internet disconnection.
3. **Privacy & Data Sovereignty Risks (`DPDP Act 2023`)**: Citizens fear sharing sensitive PII (`Aadhaar 12-digit UID, Bank Account numbers, IFSC codes`) with cloud chatbots that could log, cache, or leak confidential identity vectors.
**Aawaaz (`आवाज़`)** solves this through a **Zero-Trust Hybrid Edge-to-Cloud AI Architecture** that combines real-time multi-modal voice processing with strict **On-Device PII Shielding** and **Intelligent Live DOM Autonomous Form Filling**.
---
## 🏛️ Senior Architect System Design & Core Innovations
```mermaid
graph TD
User["🎙️ Rural Citizen (V