srikaya jam
NOVA (Navigable Open-World Virtual Agent) is an autonomous agent built on Mistral that can reason about goals, navigate complex 3D environments, and execute multi-step plans in dynamic virtual worlds.
Core trick
Serializes 3D environment state to text so a LoRA-fine-tuned Mistral model emits hierarchical action plans, retrained over three rounds of its own rollouts.
Tags
Description
NOVA (Navigable Open-World Virtual Agent) is an autonomous agent built on Mistral that can reason about goals, navigate complex 3D environments, and execute multi-step plans in dynamic virtual worlds. Instead of mapping instructions directly to actions, NOVA converts structured environment states into textual representations and uses a Mistral-based planning module to generate hierarchical action sequences. This project demonstrates how general-purpose language models like Mistral can serve as structured decision-making engines for embodied AI systems, with applications extending beyond games to robotics, simulation, and autonomous agents operating in real-world environments.
README excerpt
# NOVA — Navigable Open-World Virtual Agent **Mistral AI Worldwide Hackathon 2026 — Fine-Tuning by Weights & Biases Track** Fine-tuning Mistral Small 3.1 (24B) with LoRA across 3 self-improvement generations on structured navigation tasks. Success rate improves from **4% (zero-shot baseline) to 84%** after generation 3. Average episode length drops from **49.1 to 14.6 steps**. --- ## W&B Report Full training report with loss curves, eval metrics, action distributions, ablations, and failure analysis: **https://wandb.ai/leadgen12344-nanyang-technological-university-singapore/nova-planner/reports/NOVA:-Self-Improving-Navigation-Agent-via-LoRA-Fine-Tuning--VmlldzoxNjA2ODk1MQ==** W&B Project (all runs + artifacts): **https://wandb.ai/leadgen12344-nanyang-technological-university-singapore/nova-planner** --- ## Results | | Baseline | Gen 1 | Gen 2 | Gen 3 | |---|---|---|---|---| | Success rate | 4% | 31% | 58% | **84%** | | Avg steps to goal | 49.1 | 37.8 | 27.2 | **14.6** | | Token accuracy | — | 61% | 79% | **93%** | | Dataset samples | 0 | 312 | 694 | 1,189 | | Collection success | 0% | 18% | 43% | 71% | --- ## W&B Track Checklist This submission addresses every judging criterion for the Fine-Tuning by Weights & Biases track: ### Technical Quality - Fine-tuning is non-trivial: zero-shot Mistral Small 3.1 achieves only 4% — prompt engineering is insufficient - Full pipeline: data collection, trajectory filtering, LoRA fine-tuning, evaluation, self-improvement loop