Skip to content

The Dark Knight

▎ Sketch2App turns hand-drawn UI sketches into working web apps via a multi-step Gemini agent.

Code Video Cerebral Valley

Core trick

Sends a canvas sketch to a vision model with a JSON schema for the plan, then generates one self-contained HTML file for a sandboxed iframe.

Tags

dev toolsvision

Description

▎ Sketch2App turns hand-drawn UI sketches into working web apps via a multi-step Gemini agent. ▎ ▎ The problem: the fastest path from "idea in your head" to "working product on screen" today is typing a 200-word prompt. That's a writing skill, not a design skill. ▎ Sketching is universal — PMs, designers, founders, and kids all sketch. We make the sketch the prompt. ▎ ▎ A user draws a wireframe on a canvas. A Gemini-powered agent then runs three phases: ① Analyze the sketch using vision + structured JSON output (responseSchema) to ▎ produce a typed Plan — title, components, theme, interactions. ② Build a complete self-contained HTML/CSS/JS page from the plan + sketch. ③ Refine in a chat loop — the ▎ user types "make it dark mode" or "add a Google sign-in" and the agent patches the live app in place. ▎ ▎ The whole thing is one HTML file. No backend, no build step, no install. The API key stays in the browser; the generated app runs in a sandboxed iframe. Multimodal ▎ reasoning is the creative tool — vision in, working code out, no intermediate text representation the user has to author. --- Public GitHub Repository *
gangsd92/sketch2app★ 10 forksHTMLCreated May 23, 2026 · last push May 24, 2026

README excerpt

# Sketch2App

**Sketch a UI. A Gemini agent analyzes, plans, builds, and chats with you to refine it — live.**

Built for the Google I/O Hackathon 2026.

![demo](https://img.shields.io/badge/Gemini-2.5%20Flash%20%2F%20Pro-8b5cf6) ![agent](https://img.shields.io/badge/multi--step-agent-06b6d4) ![multimodal](https://img.shields.io/badge/multimodal-vision%20%E2%86%92%20code-10b981) ![zero install](https://img.shields.io/badge/zero-install-fafafa)

---

## What it does

A three-pane studio with a Gemini-powered agent in the middle:

1. **Sketch** — draw a rough wireframe on a canvas (left pane).
2. **Agent** — hit *Run agent*. A multi-step Gemini agent:
   - **① Analyze** — vision model reads the sketch and produces a **structured plan** (JSON-schema enforced): title, summary, components, theme, interactions.
   - **② Build** — composes the plan + sketch into a complete, self-contained HTML page.
   - **③ Refine** — chat box at the bottom of the agent pane. Type "make it dark mode", "add a sign-up link", "use a teal palette" — Gemini patches the live app. **You can even draw on the sketch to point at things or add new elements while you chat.**
   - **④ Voice** — hit the microphone icon to dictate your refinements hands-free.
3. **Live output** — the generated app runs in a sandboxed iframe on the right. Toggle to Code view, copy, or download.

Built with **Tailwind (shadcn-style aesthetic)**, vanilla JS, one HTML file, no backend.

## Why it matters

The fastest path from "idea

Team