Skip to content

strawhats

an Electron-based desktop overlay that acts as a "digital friend" for non-technical users navigating complex government websites.

Code Video Cerebral Valley

Core trick

Streams screen captures and voice into Gemini Live, which draws on-screen highlights and performs clicks only after a spoken yes.

Tags

accessibilityagentsvision

Description

an Electron-based desktop overlay that acts as a "digital friend" for non-technical users navigating complex government websites. By using Gemini Multimodal Live, the app "sees" the user's screen through continuous captures and "hears" their voice to provide real-time guidance. Instead of just talking, it draws visual arrows and highlights directly on the screen to show exactly where to click and can even perform actions (like clicking or typing) once the user gives a voice "yes" to confirm. This combines AI vision with human-in-the-loop automation to turn a confusing tax or ID portal into a simple, guided conversation.
Keshavkumar04/screen-assit★ 00 forksJavaScriptCreated Feb 14, 2026 · last push Feb 14, 2026

README excerpt

# Screen Agent

A desktop AI assistant with a floating bubble UI, voice-in/voice-out interaction, continuous screen watching, and semi-autonomous agent actions (highlight, click, type, scroll).

Powered by Google Gemini 2.0 Flash Live API.

## Key Features

- **Floating Bubble UI**: Minimal 60px draggable bubble that expands to a popup menu
- **Voice Interaction**: Speak naturally and get voice responses from the AI
- **Screen Watching**: Captures screenshots every 5 seconds for visual context
- **Visual Overlay Highlights**: AI draws pulsing highlights on your screen to guide you
- **Agent Actions**: AI can click, type, and scroll on your behalf (with confirmation)
- **Multi-Language**: English, Hindi, Spanish, French, German, Japanese, Korean, Chinese
- **Cross-Platform**: Windows, macOS, Linux

## Setup

1. Get a Gemini API key from [Google AI Studio](https://aistudio.google.com/apikey)
2. Install dependencies:
   ```bash
   npm install
   ```
3. Run the app:
   ```bash
   npm start
   ```

## Usage

1. Click the floating bubble to open the menu
2. Go to **Settings** and enter your Gemini API key
3. Click **Start Session** to begin
4. Grant screen capture and microphone permissions when prompted
5. Speak to the AI - it sees your screen and responds with voice
6. The AI can highlight areas, click buttons, type text, and scroll for you

## Architecture

```
Main Process (index.js)
  ├── Window Manager (window.js) - Bubble + Overlay windows
  ├── Gemini Integration (gemini.js

Team