ReplicaLab
ReplicaLab is an OpenEnv-based multi-agent environment for evaluating and training AI systems to design realistic replication and experiment plans under constraints.
Core trick
A Judge scores Scientist-agent plans on rigor, feasibility, fidelity, and parsimony while a Lab Manager agent enforces budget and schedule limits.
Description
ReplicaLab is an OpenEnv-based multi-agent environment for evaluating and training AI systems to design realistic replication and experiment plans under constraints. A Scientist agent proposes a plan, a Lab Manager enforces limits like budget, compute, schedule, and tools, and a deterministic Judge scores rigor, feasibility, fidelity, and parsimony. It solves the gap between ideal research plans and what can actually be executed in the real world. Users include AI researchers, ML engineers, eval teams, benchmark designers, and labs that want reproducible tests of planning, negotiation, and constraint reasoning. It can be improved with more domains like biology and chemistry, stronger paper-to-scenario ingestion, better trained policies, richer observability, and more human-in-the-loop review.
README excerpt
--- title: ReplicaLab emoji: "🧪" colorFrom: blue colorTo: green sdk: docker app_port: 7860 pinned: false --- # ReplicaLab **A multi-agent constraint-aware planning environment built on [OpenEnv](https://github.com/openenv)** > *Over 70% of landmark studies fail to replicate. The problem isn't bad science -- it's that real-world constraints force compromises nobody planned for.* ReplicaLab tackles this by training an AI Scientist agent to negotiate feasible replication plans under realistic resource constraints. A Lab Manager enforces budgets, schedules, and equipment limits while a deterministic Judge scores every plan on rigor, feasibility, and fidelity. Through reinforcement learning, the Scientist learns to ask better questions, make smarter tradeoffs, and reach agreement faster -- all without sacrificing scientific quality. Three scenario families ship today -- mathematics reasoning, ML benchmark replication, and offline finance/trading backtest design -- each with easy, medium, and hard difficulty scaling. Physics and biology remain future adapters after the core normalized scenario layer is stable. ## Team Ownership | Owner | Current focus | |------|----------------| | Kian (Person A) | Shared schemas, validation, scenario engine, judge logic | | Person B (Ayush) | Scientist prompting and parsing, notebook and client path | | Max (Person C) | Server, deployment, and runtime plumbing | | Kush (Person D) | Frontend, UI polish, docs, and demo assets | --- ## Archi