nsega
The project "RouteCause" is an autonomous diagnostics agent for LLM inference infrastructure on Kubernetes.
Core trick
Parallel subagents each pull one evidence source (metrics, scheduler config, endpoint state), then a verifier subagent tries to falsify each hypothesis.
Description
The project "RouteCause" is an autonomous diagnostics agent for LLM inference infrastructure on Kubernetes. When shared model-serving degrades, it investigates and hands platform/infra engineers a root-cause report with a concrete fix — turning a slow, manual, four-source correlation into one automated run.
The problem: LLM serving on Kubernetes (Gateway API Inference Extension / InferencePool / Endpoint Picker (EPP) scheduling, now llm-d Router) usually degrades because of routing and scheduling misconfiguration, not capacity. Symptoms — P95 Time to First Token(TTFT)/latency breaching SLO, queue depth piling on some replicas while others idle, KV-cache saturation, prefix-cache hit-rate collapse after a rollout — require cross-referencing gateway/EPP metrics, scheduler config, endpoint health, and recent manifest diffs. When you're paged on an SLO breach, that correlation is slow and error-prone.
How it works: an SLO breach (or a manual POST /diagnose) triggers the agent. Context-isolated subagents each pull one evidence source — Prometheus metrics, the EPP ConfigMap, and live cluster/endpoint state — and form hypotheses independently, so no single context owns all the evidence. Each surviving hypothesis then faces an independent verifier subagent whose job is to falsify it. The output is a single root-cause report: the diagnosis, ≥2 pieces of cross-source evidence, and a fix as a manifest diff or "kubectl patch" validated with "kubectl apply --dry-run=server". It targets three fault classes — EPP scorer-weight misconfig, an unhealthy endpoint left in rotation, and prefix-cache routing silently disabled by a rollout. Stretch goal: apply the fix and confirm recovery against the SLO.
Scope and provenance: the agent was built today in "nsega/routecause". The Kubernetes test environment (kind cluster + llm-d Router + InferencePool + simulated backends + fault injection) is a pre-existing project, "nsega/inference-lab", brought in as scaffolding and referenced, not rebuilt — see PROVENANCE.md for the build-today vs. brought-in boundary. The deliverable is the agent and its reports; there is no metrics dashboard.
README excerpt
# RouteCause
**Autonomous inference-infrastructure diagnostics agent for llm-d Router (EPP) gateways on Kubernetes.**
You get paged on an SLO breach for a shared LLM inference gateway. The cause is
rarely capacity — it's routing/scheduling: a mis-weighted EPP scorer, an
unhealthy endpoint left in rotation, prefix-cache-aware routing silently
disabled by a rollout. Diagnosing it today means hand-correlating gateway
metrics, scheduler plugin config, endpoint state, and recent manifest diffs.
RouteCause does that correlation autonomously and hands back a **root-cause
report**: root cause, supporting evidence (specific metrics + config lines), and
a concrete fix (a `kubectl patch` / manifest diff validated by
`kubectl apply --dry-run=server`) with expected impact.
> The product is the **agent and its report**, not a dashboard. The live view is
> a single RCA report.
## How it works
`POST /diagnose {pool}` runs a quarantined multi-agent inner loop:
1. **Evidence collection** — three isolated-context agents read *disjoint*
sources so no single context owns all the evidence (avoids self-preferential
bias): **E1** Prometheus metrics, **E2** the `epp-config` scheduler config,
**E3** Kubernetes cluster state (endpoint health, pool membership, arg diffs).
2. **Hypothesis merge + ranking.**
3. **Verification** — one independent refuter per surviving hypothesis, in a
fresh context, mandated to *falsify* it.
4. **Report + fix** — a fix validated with `kubectl apply --dry