Skip to content

Aria

Aria is a music-reactive AI companion that vibes with both you and your music.

Code Video Cerebral Valley

Core trick

Pre-analyzed song beat and energy data drive Gemini-generated dance choreography for a VRM avatar that tracks the user's face.

Tags

creative/musicvisionvoice

Description

Aria is a music-reactive AI companion that vibes with both you and your music. She's a 3D avatar that listens to your music, watches your face, and reacts in real-time with dancing, emoting, chatting about what you're hearing.
photon-cat/musiccompanion★ 00 forksTypeScriptCreated Feb 28, 2026 · last push Feb 28, 2026

README excerpt

# Aria - AI Music Companion

An interactive AI music companion featuring a 3D anime avatar that dances to music in real-time, tracks your facial expressions, and chats with you via text or voice.

![Built with Next.js](https://img.shields.io/badge/Next.js-15-black) ![Python](https://img.shields.io/badge/Python-FastAPI-blue) ![Three.js](https://img.shields.io/badge/Three.js-VRM-green) ![Gemini](https://img.shields.io/badge/Google-Gemini_2.5-orange)

## Features

- **Music-Synced Dancing** - Avatar performs choreographed dances synchronized to beat, energy, and mood of the music. Songs are pre-analyzed with librosa and choreography is generated by Gemini.
- **3D VRM Avatar** - Anime-style avatar rendered with Three.js, supporting blend shape expressions, skeletal animations (VRMA), and procedural dance motions (hip bounce, arm swings, chest twists, etc).
- **Real-Time Face Tracking** - MediaPipe captures your head pose, smile, blinks, and brow movements, and mirrors them onto the avatar.
- **Voice Chat** - Speak to Aria in real-time via Gemini Live API with 26+ voice options. She responds with synthesized speech and animated reactions.
- **Text Chat with Tool Use** - Gemini-powered conversation with function calling to trigger animations, change expressions, and play music.

## Architecture

```
Browser (Next.js + Three.js)          Server (FastAPI)
┌─────────────────────────┐           ┌──────────────────────┐
│  AvatarPanel (VRM/VRMA) │◄──────────│  POST /api/chat      │
│  A

Team