Orate
Orate is a library for Programmatic Grammar over Inference of LLMs.
Core trick
Compiles decorated Python generators into grammars whose field predicates constrain decoding, recompiling when earlier fields change.
Description
Orate is a library for Programmatic Grammar over Inference of LLMs. It unifies structured output, toolcalling under one abstraction, @program, which headline feature is that it allows you to constrain the fields you extract with arbitrary python predicate. Instead of only typing, now we can also guide the output with logic.
The library handles compiling your decorated program schema into a grammar, and how it should recompile when needed depending on how the execution goes (as fields can reference other fields!). These programs are also composable, and it then also handles switching grammars. Today only rudimentary optimizations are in place, like grammar caching so we don't always need to recompute.
README excerpt
# orate
**Programmatic grammar decoding for local LLM inference.**
> 🎬 **Watch the 3-min submission video:** [`video/orate_submission.mp4`](video/orate_submission.mp4)
>
> 📖 **Read the in-depth writeup:** [`docs/project_overview.md`](docs/project_overview.md)
> Structured output constrained the shape.
> Tool calling constrained the side effect.
> **orate lets the model enforce the legality of its own thought.**
You write a Python generator. At every `yield`, the model proposes; orate's
grammar + your `where=` predicate decide what's allowed. Types, tool calls,
and control flow are all the same yield stream. No JSON-mode fallback, no
post-parse, no XML scaffolding — the runtime constrains the next token at
the logit level.
> Pre-alpha. Built for *Built with Opus 4.7: a Claude Code hackathon*
> (Apr 21–26, 2026). Problem statement **#2 — Build For What's Next**.
---
## The one primitive
```python
from orate import program, gen
@program
def two_digit_prime_with_digit_sum_10():
n = yield gen.integer(
10, 99,
where=lambda v: is_prime(v) and digit_sum(v) == 10,
)
return n
```
Every `yield gen.X(...)` is a decision point. The grammar argument
constrains *shape*; the `where=` predicate constrains *value*. On
predicate reject, the engine tightens its accept set and re-samples from
the same KV — never returns a value the predicate didn't approve.
---
## Install
```bash
python3 -m venv .venv
.venv/bin/pip install -e ".[dev]" # kernel +