Upskiller
Upskiller is an RL environment that trains LLMs to agentically invoke skills — deciding which to load, when to unload, and when to submit under a context budget.
Core trick
Rules-based deterministic rewards score skill load, unload, and submit decisions under a context budget, trained with GRPO and LoRA.
Tags
Description
Upskiller is an RL environment that trains LLMs to agentically invoke skills — deciding which to load, when to unload, and when to submit under a context budget. SkillsBench shows skill invocation fails over half the time; we built Upskiller to close that gap.
The environment uses synthetic, fictional skills that can't exist in training data, forcing generalization. Tasks are dynamically generated to prevent memorization. Rewards are deterministic and rules-based — code execution, structural verification, multi-part checks, no LLM judge. Our 5-signal reward trains correctness, precision, recall, context hygiene, and token efficiency.
Trained with GRPO + LoRA (r=128) across Qwen 3 1.7B, 4B, and 8B using HF TRL. Built on OpenEnv 0.2.1, deployed on HF Spaces.
README excerpt
--- title: Skill Invocation Environment colorFrom: indigo colorTo: gray sdk: docker pinned: false app_port: 8000 base_path: /web tags: - openenv --- # Skill Invocation Environment An OpenEnv RL environment that trains LLMs to make better decisions about **when to invoke procedural knowledge (skills)** during task-solving. ## Why This Matters SkillsBench showed that AI agents fail to invoke available skills ~56% of the time, even when skills would significantly help. This environment creates a training ground for this specific problem. ### When Skills Are Irreplaceable Skills are essential when the task requires knowledge that: 1. **Cannot be derived from general training data** (e.g., proprietary API authentication protocols) 2. **Has precise, non-obvious specifications** (e.g., binary format byte layouts, exact CLI commands) 3. **Would be impossible to guess correctly** (e.g., specific error code formats, deployment phase configurations) ## Context Cost Model Skills aren't free — each loaded skill consumes context budget. The environment rewards precision: agents that load only the skills they need get higher rewards than agents that load everything. ### Actions - `load(skill_id)` — Load full skill content into context (costs budget) - `unload(skill_id)` — Remove skill from context (frees budget) - `submit(answer)` — Submit solution (reward computed on loaded state at submit time) The skill catalog (short descriptions) is returned in every observation, so agents