Overturn

Inspiration Her cardiologist already wrote the proof down. It is in her chart, in his words: "Technically limited study... cannot be quantified... the question raised clinically is not answered by this study." She is 83. He suspects cardiac amyloidosis, a disease where waiting is itself the damage, so he ordered the one scan that can confirm it. The insurer said no. Their letter states that her echocardiogram was adequate. It was not, and her own cardiologist had already said so, in the same file their reviewer was holding. Someone at the clinic could fight this. They would pull up the insurer's published policy, walk her chart criterion by criterion, and write a letter quoting the insurer's own rules back at them by section number. Two hours of careful work. The person who would do it has forty other things waiting this afternoon. So the letter does not get written. Of roughly 85 million in-network claims denied in 2024, 262,982 were appealed. Fewer than one in three hundred, and insurers upheld 66% of even those ( KFF, 2024 ). That gap is not a failure of medicine or of merit. It is letters nobody had time to write. Labour is the part a machine can take over. (She is invented. Synthea data, a fictional payer, an authored chart. That is the only merciful thing about the story, and it does not change the 85 million.) What it does The demo video shows the system working. This write-up is about what it cost to make those three minutes honest. Overturn is seven agents that read a denial letter, find the payer's governing policy, check the chart against it criterion by criterion, draft an appeal citing sections by identifier, verify every citation before a human sees it, and then hold the case for weeks, escalating on its own when the payer's deadline passes in silence. Two signatures release it: a billing clerk for the paperwork, the ordering clinician for the medicine. Nothing transmits until both are present on the same draft. The part worth stating plainly is what it refuses to do. Of the eight cases in the harness, four correctly produce no appeal at all : one quarantined document, and three declines where either no policy in the corpus governs the denial or the chart is silent on the criterion the payer actually cited. On the review queue, three of eight cases sit in a refusal state, and the screen says so in those words. This matters more than it sounds. A system that appeals everything is worse than useless to a clinic, because it spends the clerk's scarce afternoon on letters that were always going to lose. The hard cases are not the ones with no evidence. They are the charts that document seven criteria beautifully and are silent on the eighth, which is the one the payer actually denied on. No count of satisfied criteria separates those from a real appeal. Both of ours decline, and both tell the clerk which note to go and get instead. Overturn never decides whether care is appropriate. It decides whether the chart already satisfies rules the payer already published. That distinction is enforced structurally, not by instruction: no agent can assert a clinical fact that does not trace back to a row in the criteria matrix. How we built it Gemini on Vertex AI ( gemini-3.5-flash and gemini-3.7-flash ), with gemma-4-26b-a4b-it-maas guarding the front door, on Google ADK, Cloud Run, Pub/Sub, Firestore, Cloud Scheduler, Cloud Storage, Model Armor, and OpenTelemetry into Cloud Trace. That list is the least interesting thing here. Three decisions did the real work. State lives in Firestore, not in a process. Once an appeal is filed, the case sits for weeks with nothing executing. No loop, no open connection, no object holding it together in memory. A scheduled job wakes up, queries one timestamp field, and a worker that has never seen the case reconstructs it from a single document and continues. That is the entire difference between multi-week operation and a loop with a sleep in it, and it is why a restart, a redeployment, or a cold start costs nothing. Every outward action passes an idempotency guard. Pub/Sub delivers at least once, which means your handler will run twice. Without a create-only write keyed on (case_id, action_type, attempt) , a redelivery files the same woman's appeal a second time. Claims carry leases, so a worker that dies mid-action does not block that action forever, and the takeover is recorded rather than silent. Permissions are enforced twice, for two different reasons. IAM genuinely scopes buckets, topics, and secrets per identity. Firestore has no collection-level IAM, so a deterministic gateway enforces the same matrix in code, and every datastore consumer routes through it. Drafting holds no read on the policy corpus, so it cannot go looking for supporting material Mapping did not hand it. Verification holds no write on cases, so it cannot edit the draft it is judging. Neither is a rule a developer has to remember. The handle an agent receives will not perform the call. All data is synthetic: Synthea patients, encounters we authored, and an invented payer whose six policies (42 sections, 113 citable identifiers) follow the structure real payers publish openly. Challenges we ran into Our own gateway had four doors we did not know about. The module docstring said every datastore consumer routed through it. Four did not, including one in the same file the docstring named specifically. One of them was writing to quarantine under an identity that holds only READ on that collection, and it worked precisely because it bypassed the check that would have refused it. The lesson was not that we had a bug. It was that a security boundary nobody has tried to walk around is a claim, not a boundary. We found the doors only once we started treating our own documentation as a hypothesis. A page that passed the entire test suite returned 500 in production. /fleet reads the agent roster from infra/agents.env at request time, and infra/ is excluded from the container image. Locally the repository is the filesystem, so nothing caught it, and we had just changed the front page to link straight there. Our tests now compare what the application opens at runtime against what the Dockerfile actually ships. A deploy took the entire site down twelve hours before the deadline. gcloud run deploy --no-allow-unauthenticated does not merely decline to add the public binding. It removes an existing one . Our deploy script then granted the operator's own account instead. That binding is useless in a browser, because Cloud Run wants a bearer token and Chrome does not send gcloud credentials. So every judge-facing URL returned 403, including the architecture diagram, while the deploy log reported success in green and the operator, testing from the command line, saw nothing wrong. deploy.sh now curls all five public paths after every deploy and refuses to end quietly on a failure. The lesson generalises well past this project: "deployed successfully" and "reachable by the people it is for" are different assertions, and only one of them was being tested. Accomplishments that we're proud of The verifier catches things nobody told it to look for. Running live, Drafting described a CASE-001 encounter as a "telehealth evaluation." The chart calls it an "interim review" and never says how it was conducted. Verification, a separate agent on a separate model with no ability to edit the draft it is judging, killed it unprompted, fed the specific objection back as revision instructions, and attempt 2 passed. It then caught the identical overclaim again on a separate run two days later. Reproducing is what separates a finding from a lucky sample. Fabrication is refused in Python, not by a model. When Drafting invented policy section NBH-CARD-014-9.9 , the existence check rejected it as set membership against the retrieved identifiers. Zero model calls spent, because a model cannot be wrong about a question it is never asked. The measurement harness does not trust the thing it measures. Our evaluator re-derives every citation and every chart quote independently rather than asking the verification layer whether it did well. A harness that trusts the component it is grading measures nothing. It is cheap enough to be real. All eight cases run end to end in 44 model calls and roughly 266,000 tokens, about eleven calls per appeal actually produced. What we learned Everyone says defence in depth. It only means anything when the layers fail independently , and we ended up with evidence in both directions. Google's own guardrail missed an attack ours caught. We fed the pipeline a denial letter carrying a hidden instruction telling whatever read it to approve the claim and forward the chart elsewhere. Model Armor returned NO_MATCH_FOUND . All four filters, EXECUTION_SUCCESS . It is a fair miss. Read as a prompt, that letter is an ordinary business document containing some imperative sentences. Our deterministic layer knows what a denial letter is shaped like, and quarantined it on seven findings. We lead with this because a negative result about a sponsor's product is worth more than another feature claim. Then the same architecture failed in the opposite direction, and that one cost more. On the deployed run, Verification rejected CASE-003 three times and the attempt cap fired. We went back and read those rejections against the policy by hand. Two of the three were wrong. One objected to a sentence that is the criterion word for word , with "Requires that" prefixed to it. Her appeal died over paraphrasing. Because the cap fails closed, a false positive did not produce a bad letter. It produced no letter at all, which is the same outcome as never having built the system. The uncomfortable part is what that revealed about our own evaluation. Against live models on 28 August 2026 the harness reported: 8/8 fully grounded, and zero fabricated citations reaching a human 6/8 outcomes as expected, and we publish the two that were not about 11 model calls per appeal produced, across 44 calls for all eight cases What it had no measure of whatsoever is how often the checker itself is wrong. We had been reporting a safety net's catches without ever measuring its false alarms. A safety net with an unmeasured false-positive rate is half a result, and the half you cannot see is the half that costs someone their scan. We are publishing that rather than quietly re-running until CASE-003 came out clean. What's next for Overturn Measuring the false-positive rate we just admitted we cannot quantify, and softening a cap that currently fails closed on correct work. That asymmetry is the whole problem. When this system wrongly sends a weak appeal, a human reads it and catches it. When it wrongly refuses a strong one, nobody ever hears about it. The clerk never learns there was a letter worth writing, and the patient never learns an appeal was possible. Somewhere today, a letter is not getting written for a claim that would have been overturned. That is the letter this is for.