Toby
Built a system for building and governing AI agents through AR glasses.
Core trick
Agent scores its own confidence per step and escalates only uncertain actions to AR glasses, respecting user-set rules for a Bluetooth smart lamp.
Description
Built a system for building and governing AI agents through AR glasses. AI can now act on its own, but there's no clean way for a human to stay in the loop and govern an agent in the moment as it acts on the real world — you either grant full autonomy and hope, or babysit every step and lose the point. With Glance, you look at a real object, ask an agent to control it, and it builds itself, takes control, and surfaces only its genuine judgment calls to your field of view, where you resolve them with a glance or your voice — and the limits you set become rules it respects, acting on its own where you've allowed it and stopping to ask where you haven't. We prove it on a real smart lamp: you glance at it, an agent connects over Bluetooth and controls it live, proposes a schedule tuned to your day, and when told "never turn off without asking," later catches itself mid-action and asks first instead of crossing the line you drew. The whole approach is only possible because Gemini 3.5 Flash is fast and cheap enough for the agent to score its own confidence at every step — handling the routine silently and escalating only what genuinely needs you — turning "an agent that acts" into "an agent you can govern."
README excerpt
# GLANCE — autonomous-builder-with-glance-HITL
One sentence: Flash + Managed Agents + AR glasses. The agent does the routine
90% silently and surfaces only the genuine forks to the human on glass.
## Run the demo (zero credentials)
python3.13 -m venv .venv && .venv/bin/pip install -e .[dev]
npm install
bash scripts/demo-start.sh --no-adb
The hub binds to :8765 (lens WS) + :8766 (snapshot HTTP). Vite serves the
lens at :5173. Storyline walks the four acts end-to-end (Connect → Control →
Personalize → Govern+Learn) and tears down cleanly. `Ctrl-C` mid-run is
handled.
## Verify the §2 audience claim (also zero credentials)
GLANCE_USE_REAL_CLIENT=0 PYTHONPATH=src .venv/bin/pytest -q
Expect 463 passing tests. The load-bearing test is:
tests/test_v8_crossrun.py::test_v8_crossrun_escalations_strictly_decrease
It runs the autonomous builder twice on the same fixture, asserts
escalations strictly decrease (run1=3 → run2=0) driven by real
PreferenceMemory hits, NOT by a scripted decrement. Provenance hash
binds the escalation count (red-team verified).
## Architecture (the four acts)
Act 1 CONNECT — capture (glance/voice) → Flash captions → builder boots
PRs: #11 fixture, #13 capture, #14 lens-capture, #15 builder
Act 2 CONTROL — agent issues real lamp tool calls; provider reads back
PRs: #17 lamp provider, #23 builder↔lamp wiring
Act 3 PERSONALIZE — Flash proposes a daily rhythm g