Ghost
A benchmark score tells you how often a model fails.
Core trick
An adversary model generates single-condition executable perturbations to repo tasks, then compares solve outcomes before and after each change.
Description
A benchmark score tells you how often a model fails. It doesn't tell you what causes the failure, or whether the cause is real or noise. So I built a system that answers that question specifically targeting Astra.
We run Astra as a software-engineering agent on real GitHub repositories and measured its solve rate at 0.48. Then a second adversary model (Qwen as we needed open-weights) scans through the repo to find possible things to change. It proposes a small executable generator that changes exactly one condition changed. A stale tool spec. A flaky test. An underspecified issue. A truncated observation. etc
If a separate Astra model deems this a valid change (doesnt fundamentally change the objective / make it impossible) we then run evaluate Astra on the original example and the adversarially modified one. We reward the adversary based on if Astra was able to solve it originally but not with the modification and encourage diverse changes in the loss.
Overall, we produce a mapping showing which perturbations Astra typically struggles with and which it succeeds on. With the goal that this could be used to better understand the model + train a better successor.
README excerpt
# OpenAiAdversary **Find where coding agents break. Turn the evidence into training signals for stronger models.** OpenAiAdversary probes coding agents with adversarial guidance embedded in the tool responses they read while repairing real repository bugs. It compares clean and attacked runs, replays their patches against fresh tests, and captures the exact guidance, commands, code changes, and outcomes behind each finding. The interesting failures are repairs that look convincing and pass the original tests, yet break deeper behavior. Exposing that gap gives us concrete examples of what a stronger model needs to learn. ## What we built - **An adversarial evaluation pipeline:** executable attack programs, paired runs, fresh patch replay, and evidence checks that connect each outcome to its source artifacts. - **An agent activity demo:** a clean, expandable feed that walks through task context, attack generation, paired repairs, verification, and findings. Recorded evidence drives the presentation; you can inspect earlier activity while the sequence continues. - **An interactive results explorer:** browse failure patterns and successful repairs, compare clean and attacked patches, inspect exact injected guidance and model traces, and download the supporting evidence. - **A concise final report:** summarizes where Astra struggled, where it succeeded, and how those examples could inform training and evaluation of the next model. The included dataset contains **147 evaluate