Sideman
Sideman is an AI‑powered web app that teaches jazz piano comping and improvisation using real lead sheets.
Core trick
Extracts chords from an uploaded lead-sheet image with Gemini Vision, then matches microphone audio against each target voicing.
Description
Sideman is an AI‑powered web app that teaches jazz piano comping and improvisation using real lead sheets. Users upload a PDF or image of any jazz tune, and Gemini automatically extracts the chords, modes, form, and suggested voicings. The system then generates a personalized, bar‑by‑bar lesson.
Users practice each chord by playing into their microphone. Gemini detects shell voicings, rootless voicings, drop‑2 shapes, and extensions. It provides real‑time pedagogical feedback on voicing choice, accuracy, voice‑leading, and improvisation modes.
The app's next steps will include dynamically generated piano chord diagrams using Nano Banana Pro to highlight the correct notes on a provided keyboard image. Lessons will adapt to the user’s performance, and progress is tracked across sessions.
This project brings together advanced audio analysis, AI music understanding, and interactive visual teaching to create a modern, highly flexible jazz learning tool.
README excerpt
# Sideman
An AI jazz piano teacher for the browser: upload a lead sheet, get the chords extracted, and practice voicings with a piano diagram and microphone chord recognition.
[](https://github.com/sarpvulas/Sideman/actions/workflows/ci.yml)
[](LICENSE)
## TL;DR
Learning jazz comping means translating chord symbols into voicings and checking your own playing by ear. Sideman reads a lead sheet with Gemini Vision, turns each bar into a practice target, and listens through the microphone to tell you whether you played the chord. It is an early-stage prototype: the core flow runs end to end, but progress is not persisted.
> TODO(sarp): add screenshot after deploy
## What it does today
Verified in the code and by running the API routes locally:
- **Score analysis.** Upload a PDF, PNG or JPEG (max 4 MB). `POST /api/score/upload` sends it to Gemini (`gemini-2.5-pro`) and returns title, key, time signature and per-bar chord symbols, voicing notes and a suggested improvisation mode. Without an API key it serves a built-in sample lead sheet, flagged `isSample` in the response and labelled "Sample score" in the UI. With a key set, a failed analysis returns 502 ("Could not analyze this lead sheet...") instead of the sample.
- **Lessons.** Start a lesson from an analysis and step through the bars with a piano keyboard diagram.
- **Micropho