Finalist
RPG - Robotic Planning with Gemini
Deploying robots in real-world indoor spaces is fundamentally constrained by two problems: environments are unknown ahead of time, and robot morphologies dictate what actions are physically feasible.
Core trick
Gemini builds a spatial map from sequential photos, then plans tasks with embodiment-specific action constraints enforced at each stage.
Description
Deploying robots in real-world indoor spaces is fundamentally constrained by two problems: environments are unknown ahead of time, and robot morphologies dictate what actions are physically feasible. A humanoid can reach a high shelf; a quadruped cannot. A mobile base can sweep an open floor; it cannot navigate a cluttered corridor. Traditional planning stacks require pre-built maps, hand-coded action libraries, and per-robot engineering. RPG eliminates all three by using a single multimodal model (Google Gemini) to perceive, map, and plan — with embodiment-specific constraints enforced at every stage.
The system operates entirely from photographs. No LiDAR, no SLAM, no prior map data. A user walks through an unseen room, uploads sequential photos, and RPG synthesizes a structured spatial understanding, a generated floor plan, and a validated task plan — all conditioned on the selected robot morphology.
README excerpt
# RPG — Robotic Planning with Gemini Zero-shot task planning for heterogeneous robot embodiments in previously unseen indoor environments, powered by multimodal foundation models. --- ## Motivation Deploying robots in real-world indoor spaces is fundamentally constrained by two problems: environments are **unknown ahead of time**, and robot morphologies **dictate what actions are physically feasible**. A humanoid can reach a high shelf; a quadruped cannot. A mobile base can sweep an open floor; it cannot navigate a cluttered corridor. Traditional planning stacks require pre-built maps, hand-coded action libraries, and per-robot engineering. RPG eliminates all three by using a single multimodal model (Google Gemini) to perceive, map, and plan — with embodiment-specific constraints enforced at every stage. The system operates entirely from photographs. No LiDAR, no SLAM, no prior map data. A user walks through an unseen room, uploads sequential photos, and RPG synthesizes a structured spatial understanding, a generated floor plan, and a validated task plan — all conditioned on the selected robot morphology. --- ## Core Capabilities ### Planning in Unseen Environments RPG treats every environment as unknown. There is no reliance on pre-existing maps, object databases, or spatial priors. The full perception-to-plan pipeline runs from raw imagery: 1. **Scene decomposition** — Sequential photographs are encoded and sent as a single multimodal prompt to Gemini. The model r