Skip to content

Crossmodal Search Engine

A natural-language search tool for the Waymo Motion Dataset.

Code Video Cerebral Valley

Core trick

Captions 8-camera mosaics with a vision model, embeds captions into a local vector index, and serves natural-language frame search.

Tags

visionagentsdev tools

Description

A natural-language search tool for the Waymo Motion Dataset. Using Waymo’s multimodal data—camera frames, object positions, acceleration, trajectory, and scene context—the system lets users describe the driving scenario they want to find in plain language. For example: “Find scenes where a pedestrian jaywalks.” The tool then searches across the dataset and returns the exact frames or scene segments that match the requested situation, making it easier to identify complex behavioral patterns without manually reviewing hours of autonomous driving data.
jin-dalrae/2605-GoogleIO-crossmodal-search★ 00 forksPythonCreated May 23, 2026 · last push May 25, 2026

README excerpt

# Crossmodal Search

Natural-language search over local Waymo End-to-End Camera TFRecord shards.

This repo is an MVP for indexing a local shard, captioning synchronized
8-camera frames, embedding the grounded text, and serving a search UI with a
cockpit detail page. It is not a canned demo: search results come from the
local generated index under `crossmodal_search/data/`, which is intentionally
gitignored.

## Current State

As of 2026-05-25, the implementation supports the full local path:

- scan one local E2E TFRecord shard into `frames.parquet`
- hydrate frame images from byte offsets
- build 4x2 camera mosaics
- call a managed vision captioning agent
- embed captions/tags into a local vector index
- serve a Flask search UI and frame detail page

The active local index in this workspace is still small: 20 sampled rows, with
1 real VLM-captioned frame and 19 provider failures. The real visual frame is:

```text
e0ae38e2d9675d9d889eaffcf10fe392-142
intent: GO_RIGHT
caption: A vehicle navigating an urban environment with signs of road
construction and multiple intersections, transitioning towards a curved road
section marked by retaining walls and chevron signs, under clear daytime
conditions.
```

That means visual search currently works only for concepts present in that
caption/tags. For example, `intersection` can return the real indexed frame.
`crash` returns no result because no indexed visual caption contains crash
evidence. The UI reports how many visual frames are

Team