Aria
Aria is a desktop AI agent that learns how work gets done.
Core trick
A vision model reads screenshots while a knowledge graph retrieves stored workflow trajectories, reusing matching graphs before replanning.
Description
Aria is a desktop AI agent that learns how work gets done. Instead of treating every task as a brand-new problem, Aria recognizes recurring workflows, grounds them into structured action graphs, and uses that memory to execute familiar tasks faster and more reliably over time.
We have three layers: perception, reasoning, and graph retrieval.
- Perception: A vision-language model interprets screenshots to extract the current UI state, visible affordances, and action-relevant context from the desktop.
- Reasoning: A reasoning model uses this state to plan the next step, decide when to explore, and recover from ambiguity in the environment.
- Graph Retrieval: In parallel, a knowledge-graph agent queries previously stored workflow graphs to find similar task trajectories, matching on intent, interface state, and prior action structure.
When a match workflow exists, Aria can follow or adapt an existing graph instead of replanning from scratch. When no match is found, it falls back to live reasoning and converts the new successful run into graph memory for future reuse.
README excerpt
# Zomma Zomma is a local-first desktop operator for grounded computer use. It is a product fork of the original [UI-TARS Desktop](https://github.com/bytedance/UI-TARS-desktop) project, adapted for backend-directed workflow routing, graph-aware execution, and post-run feedback. The current focus of this repo is a hackathon demo and integration path for a grounded computer-use agent: - `baseline`: normal agent behavior, no graph assist - `grounded`: backend agent decides whether to retrieve and use a graph Demo task: > Find the Uber receipt in Downloads and create an expense report spreadsheet. ## Current Scope This repo currently covers: - the Electron desktop app in [`apps/ui-tars`](./apps/ui-tars) - the Zomma renderer branding and local-first UI - integration with a backend that exposes an OpenAI-compatible VLM endpoint - optional post-run feedback capture - a local stub backend for frontend testing The desktop does not decide: - whether a workflow has been seen before - whether a query is similar enough to use a graph - whether grounded behavior should fall back Those decisions are owned by the backend. ## Product Contract Default behavior: - desktop sends the task normally - backend decides whether the workflow is new, seen, or should use graph assistance Optional demo/debug override: - desktop exposes a `Force Workflow Mode` toggle in settings - when enabled, desktop sends `X-Force-Workflow-Mode: baseline` or `X-Force-Workflow-Mode: grounded` - when disabl