SnapOn: On-Device Context-Aware Multimodal AI
SnapOn is an offline multimodal AI assistant for Android. Point your camera, speak your question, get instant answers about people, objects, documents, and scenes, powered by Qualcomm Snapdragon NPU and ExecuTorch. No cloud. Total privacy.
Core trick
Runs vision, text, and embedding models on-device through ExecuTorch on the phone's NPU, with persisted memory facts for recall.
Description
$26
README excerpt
# SnapOn **Point. Ask. Remember. All offline.** An on-device visual memory assistant for Samsung Galaxy S25 Ultra — built for hackathon June 27-28. --- ## Demo ``` [ screenshot placeholder — add docs/screenshot.png ] ``` --- ## Problem Statement We forget things constantly — where we parked, what the label says, who we met. Existing assistants send your photos and voice to the cloud, where they are processed, logged, and stored. SnapOn solves this with a **fully offline pipeline**: - **Point** your camera at anything - **Ask** a question by voice or text - **Remember** facts that persist across sessions No cloud. No account. No data leaving your device. Ever. --- ## Tech Stack | Layer | Technology | |-------|-----------| | Vision + Text + Compress + Rerank + Embed | **InternVL3-1B** (HuggingFace Transformers, CPU bfloat16) | | Speech-to-Text | faster-whisper `base.en` (CPU, INT8) | | Text Embeddings | InternVL3-1B LLM backbone (mean-pool last hidden, L2-norm) | | Vector Search | FAISS `IndexFlatIP` — cosine similarity (threshold 0.3) | | Persistent Memory | SQLite3 | | Text-to-Speech | piper-tts `en_US-lessac-medium` + sounddevice | | Planned NPU Backend | Qualcomm QNN via ExecuTorch 1.3 (target: SM8750) | | API Server | Flask + flask-cors + SSE streaming | | UI | Vanilla HTML/CSS/JS — mobile-first, camera + waveform | **One model does everything.** InternVL3-1B handles vision queries, text-only queries, memory compression, reranking, AND memory embeddings — n