evalforge
EvalForge.
Core trick
Gives Gemini function-calling tools to propose vulnerabilities, write PoCs, run them, and emit regression tests for a Python file.
Description
EvalForge. Self-improving security auditor for Python.
README excerpt
# EvalForge A self-improving security auditor for Python. Hands a target file to **Gemini 3.5 Flash** through Google's managed-agents (function-calling) runtime, lets the model **propose vulnerabilities, write working PoCs, verify them by execution, reflect on what it missed, and write regression tests** — all without us scripting the tool sequence. Built for the **Google I/O Hackathon** (1-day solo, claims the managed-agents bonus). --- ## Headline numbers (live Gemini runs) | Target | Real OSS | LOC | Confirmed bugs | Notes | |---|---|---|---|---| | `vuln-utils` (planted) | — | 105 | **7 / 7** | calibration target — 7 deliberately-planted bugs across SQLi, RCE, path traversal, deser, ReDoS, weak crypto, SSTI | | `furl` | ✓ | 1920 | **1** | real parser-differential SSRF bypass (`is_valid_host` accepts backslashes; WHATWG normalizes them) — withheld per responsible disclosure | | `python-frontmatter` | ✓ | 700 | 0 | clean | | `python-slugify` | ✓ | 250 | 0 | clean | | `xmltodict` | ✓ | 650 | 0 | clean | | `html2text` | ✓ | 1880 | 0 | 1 claim, PoC did not fire → rejected by verifier | | `python-multipart` | ✓ | 2160 | 0 | 1 claim, PoC did not fire → rejected by verifier | **Pitch:** 7/7 on the planted target. 1 real find on real OSS. **0 hallucinated findings on real OSS** — because every claim is gated on an executable PoC. --- ## Quick start ```bash # 1. Python env python3 -m venv .venv .venv/bin/pip install -e ./hero/vuln-utils pytest # 2. Frontend npm install npm