Minecraft in Real-Life: A Local AI Shelter Builder
A mobile app which uses camera scans and on-device AI to give real-time step-by-step instructions to build a simple house in the local environment without connection to the internet.
Core trick
Maps YOLO-detected objects to virtual building blocks in ARCore, and a VLM verifies each build step.
Description
$2b
README excerpt
# AI Construction Copilot > **Qualcomm × Meta ExecuTorch Hackathon** > Offline AR construction assistant — Minecraft in the real world, running entirely on-device. Point your phone at any room. On-device AI identifies real objects, maps them to virtual building materials, and overlays a step-by-step Minecraft-style build guide — no cloud, no internet, everything on the Snapdragon 8 Elite Hexagon NPU. --- ## Demo Flow 1. **Splash** — "Powered by Qualcomm AI" 2. **Scan room** — live camera, ARCore detects floor plane and shows dimensions 3. **Finish Scan** — enter the AR construction view 4. **YOLO identifies objects** — laptop → steel panel, bottle → pillar, etc. 5. **AR overlay** guides you to place objects step by step 6. **Verify Step** — on-device VLM confirms the step is complete in < 2 s 7. **Advance** — AR moves to the next step 8. **Completion screen** — build stats, start a new project --- ## Tech Stack | Layer | Technology | |---|---| | Language | Kotlin | | UI | Jetpack Compose + Material 3 | | AR | ARCore 1.41.0 (plane detection, spatial anchors) | | Camera | CameraX 1.3.1 (camera-core / camera2 / lifecycle / view) | | On-device inference | ExecuTorch + QNN backend, Snapdragon Hexagon NPU | | Architecture | Single-module MVVM, Compose Navigation | ### AI Models (both bundled in `app/src/main/assets/`) | Model | Format | Trigger | Purpose | |---|---|---|---| | YOLO mobile | `.pte` | Every frame | Detect real objects in scene | | SmolVLM | `.pte` | Tap "Ver