Sese
Sese is a multimodal AI study companion that turns problem-solving into a real-time, interactive learning experience.
Core trick
A vision model reads handwritten math from live OneNote captures, then RAG over textbook chunks grounds the voice tutor's explanations.
Description
Sese is a multimodal AI study companion that turns problem-solving into a real-time, interactive learning experience.
Students write naturally (e.g., on OneNote), and Sese observes their work, understands each step, and responds with voice guidance, structured explanations, and actionable next steps. Instead of passively consuming solutions, learners actively solve problems while Sese coaches and hypes them in real time.
Sese combines:
- Vision: understands handwritten/math input
- Reasoning: generates structured teaching steps and feedback
- Voice: explains concepts conversationally with an engaging tutor personality
- Grounding: retrieves relevant textbook content using a vector database for trustworthy explanations
The system renders a live teaching board with equations, steps, and concept notes, while the tutor speaks, corrects mistakes, and adapts to the student's progress.
This transforms studying from a static, lonely process into an interactive, guided experience, closer to having a professor available at all times.
This is the future of learning: AI that doesn't just give answers, but helps you think.
README excerpt
# Sese - AI Study Companion Sese is a real-time multimodal AI study companion. It observes a student's live handwritten work (e.g., in Microsoft OneNote) and provides guided hints, visual explanations on a digital blackboard, and encouragement through a panda companion persona. ## Features - **Voice-first onboarding**: Conversational setup for name, study topic, and optional course files. - **Three-column workspace**: Input Source · Teaching Board · Tutor Panel (unchanged layout). - **Freeform tutor chat**: Text + browser speech recognition at the bottom of the Tutor Panel; responses update the board and play ElevenLabs audio when enabled. - **Quick actions**: Hint / Explain / Check step still use the same multimodal `/api/analyze` path as frame capture. - **Textbook grounding (real RAG)**: Upload a PDF in Input → chunked → embedded with Gemini → stored in Supabase pgvector → tutor chat retrieves by similarity and shows source chips. - **Premium voice output**: ElevenLabs TTS (default energetic female “Domi”; tuning via env). - **Lightweight delight**: Confetti + streak counter on milestone completions suggested by the model. - **Optional session persistence**: When Supabase is configured, tutor turns are stored (`sese_study_sessions`, messages, board snapshots). ## Project structure - `/app/api/analyze` — multimodal frame analysis (Gemini structured JSON). - `/app/api/tutor/chat` — freeform tutor conversation + retrieval + optional last-frame image. - `/app/api/document