Florida Men
Say Hello to Tappy, it's your small buddy that lives on your phone, and can talk, teach and do tasks for you.
Core trick
Gemini Live streams voice while an Android AccessibilityService snapshots and indexes on-screen UI elements so the model can act on them.
Description
Say Hello to Tappy, it's your small buddy that lives on your phone, and can talk, teach and do tasks for you. It's built to best assist the elderly and the tech-illiterate folk.
README excerpt
<!-- SPDX-FileCopyrightText: 2026 Hari Srinivasan <harisrini21@gmail.com> --> <!-- SPDX-License-Identifier: AGPL-3.0-only --> # Tappy Live Tappy Live is a real-time Android voice assistant and phone-use teacher for Problem Statement 1: Real-Time Multimodal Interaction. It combines two layers: - Gemini Live API for low-latency, interruptible voice conversation. - A mobile computer-use layer on Android, built with AccessibilityService, app snapshots, indexed UI elements, actions, and overlays. The demo goal is to let a user summon Tappy from anywhere, ask for help in natural language, and either receive guided teaching with overlays or ask Tappy to operate apps on their behalf until a sensitive final action requires confirmation. ## Hackathon focus Tappy Live is built around Gemini Live API using: - `gemini-3.1-flash-live-preview` for real-time audio, interruption handling, and tool calls. - A Cloud Run token broker that mints short-lived Gemini Live ephemeral tokens. - Direct Android WebSocket connection to Gemini Live. - Native Kotlin Android with Compose. - Android microphone streaming through `AudioRecord`. - Gemini audio playback through `AudioTrack`. - Android AccessibilityService for the mobile computer-use execution environment. Tappy is not just a chat UI with a microphone. It sees app UI through Accessibility snapshots, reasons over indexed visible elements, can tap and type, and can draw overlays directly on top of the phone screen. ## What works now The c