Skip to content

One Piece of Data

An interactive, LLM-extracted relationship graph for the entire One Piece manga: every character, crew, organization, saga, and arc, connected by 19,320 typed edges — fought, defeated_by, ally_of, enemy_of, member_of_cre

Core trick

Extracts typed relationship triples from wiki sections, rejecting any triple whose evidence quote is under five words or absent from the source.

Tags

Description

An interactive, LLM-extracted relationship graph for the entire One Piece manga: every character, crew, organization, saga, and arc, connected by 19,320 typed edges — fought, defeated_by, ally_of, enemy_of, member_of_crew, family_of, mentor_of, and more. Existing One Piece databases store facts (bounties, appearances). They don't capture relationships — who fought whom, who's allied with whom, who's whose family — because those live in 13,000+ unstructured wiki sections. We extracted them in a single 21-minute Claude Sonnet 4.6 run for ~$91, with a verbatim evidence quote attached to every edge so the graph is auditable, not a black box. The pipeline reads One Piece Fandom wiki sections, runs them through Claude with a constrained 12-relation vocabulary and a strict "quote ≥5 words from the source or skip the triple" rule, deduplicates the 43,814 raw triples down to 19,320 unique edges, and ships them to Supabase. A React + vis-network single-page app then renders the graph live: pick any entity to focus on, expand 1-4 hops at a time, filter by confidence and relation type, and click any edge to read the exact wiki sentence that justified it. A sortable edge table below the graph lets you sort 19K rows by confidence or chapter and click straight through to that edge in the visualization. Three things make it work in production: (1) hash-based change detection so weekly re-runs after each new chapter cost ~$2 instead of $91; (2) Anthropic's structured-output JSON schema, which gave us zero parse failures across 13,119 calls; and (3) BFS-based subgraph rendering with an edge cap, so a graph with 19K edges stays interactive in the browser at ~1.5 MB gzipped. The codebase is fully reproducible from make commands and split across two AGPL-3.0 repos: a Python data pipeline and a React web app. Alongside the Story Graph we also shipped a few smaller features in the same week: a new Character Occupations page that surfaces every role a character has held (1,236 characters, 2,038 role assignments); a Chapter Release Predictor that forecasts the next 5 chapter dates from the live Shonen Jump issue schedule; devil-fruit and haki badges added to the character comparison tool and the main characters table with a column-visibility toggle and persistent table state; and shareable result images (feed and story formats) for both the Guess the Character and Who Am I? games, plus a tiered difficulty rework of the Who Am I? quiz with a recalibrated speed-based ranking system. URLs: - https://onepieceofdata.com: the web application - https://onepieceofdata.com/#/analytics/story-graph : the story graph - https://github.com/ismailsunni/onepieceofdata : the brain, the parser, the data generator repository - https://github.com/ismailsunni/onepieceofdata-react : the web app repository
ismailsunni/onepieceofdata★ 01 forksPythonMITCreated Apr 7, 2020 · last push Oct 19, 2026

README excerpt

# One Piece of Data

![One Piece of Data](./onepieceofdata-header.png)

**A modern Python data pipeline + RAG chatbot for One Piece manga data.**

One Piece of Data scrapes, validates, enriches, and stores information about One Piece
manga chapters, characters, volumes, arcs, and sagas from the
[One Piece Fandom Wiki](https://onepiece.fandom.com/). The structured data lives in
[DuckDB](https://duckdb.org/) and can be exported to PostgreSQL/Supabase. On top of it
sits a Retrieval-Augmented Generation (RAG) chatbot and an LLM-extracted story knowledge
graph.

---

## ✨ What's inside

| Capability | Description |
|------------|-------------|
| 🕷️ **Scraping** | Parallel scrapers for chapters, volumes, characters, arcs, sagas + raw wiki text |
| 📊 **Parsing & validation** | Pydantic models validate every record before it lands in DuckDB |
| 🔧 **Post-processing** | 14 enrichment steps: birth dates, affiliations, devil fruits, haki, occupations, appearance analytics, importance scoring |
| 🗄️ **Storage** | DuckDB as the analytical source of truth; PostgreSQL / Supabase as export targets |
| 🤖 **RAG chatbot** | Groq LLM with tool-calling over SQL + vector search of embedded wiki text |
| 🕸️ **Knowledge graph** | LLM-extracted entity/relationship triples (`graph_nodes`, `graph_edges`) |
| 🌐 **Network explorer** | Local interactive web app for the character co-appearance network |
| ⚙️ **Tooling** | `uv` for deps, `make` for orchestration, `ruff`/`black` for quality, `pytest` for test

Team