SenseBridge
Accessibility of communication for differently able people is a challenge.
Core trick
Routes each participant's speech through Gemini Live into captions for deaf users and synthesized speech for blind users in one shared audio room.
Description
Accessibility of communication for differently able people is a challenge. In an evolving digital world, we need a radically different approach compared to the presently available solutions. Which is why we present SenseBridge, it allows deaf, blind and mute people to effectively communicate with each other. While addressing their special needs. Deaf people need captioned audio, blind people need speech and mute people need to be able to convert text to speech in order to communicate. Our solution tries to bridge this gap through the use to Gemini models for speech to text, text to speech and vision models.
The app was hosted on a cloudflare tunnel for the demo purposes.
README excerpt
# SenseBridge Live **A shared, real-time audio meeting room that bridges blind/low-vision and deaf/hard-of-hearing users, powered by the Google Gemini Live API.** Built for **Problem Statement 1: Real-Time Multimodal Interaction (Gemini Live API / Live Translate)**. --- ## Problem statement Communication between a **blind or low-vision** person and a **deaf or hard-of-hearing** person is uniquely hard: one cannot easily read text or see gestures, the other cannot easily hear speech. Existing tools solve one direction (speech-to-text *or* text-to-speech) but rarely bridge both sides in a single, real-time conversation. **SenseBridge Live** is a two-person **meeting room** (audio only, no video) where each person joins with a room code and gets a UI tailored to their needs: 1. The **blind user speaks** → their words are captioned live for the deaf user. 2. The **deaf user reads** the captions and **types or taps a reply**. 3. That reply is **spoken aloud automatically** on the blind user's device. 4. Either side can request a **slower, clarified** version of the last instruction. 5. Either side can generate a **text + spoken recap** of the whole conversation. 6. The deaf user can use **Scene Assist** (camera) to get a spoken description of nearby signs / surroundings. Real audio also streams peer-to-peer over WebRTC, so it is a genuine call — not just a shared screen. --- ## Features | # | Feature | What it does | |---|---------|--------------| | 1 | **Role-based roo