Skip to content

tres-gemmas

This Gemini extension exists to make it effortless to run high-performance, on-device machine learning models.

Code Video Cerebral Valley

Core trick

Node.js native bindings wrap the LiteRT-LM C++ runtime, exposing on-device model inference on CPU or GPU through an mcp-llm-compatible API.

Tags

dev tools

Description

This Gemini extension exists to make it effortless to run high-performance, on-device machine learning models. It packages the LiteRT-LM runtime, GPU-first bindings, and MCP-compatible tooling into a single, hackathon-friendly bundle so teams can focus on building experiences rather than wiring up infrastructure.
maceip/gemma3-mcp★ 20 forksJavaScriptCreated Nov 9, 2025 · last push Apr 20, 2026

README excerpt

# tres vibes: accelerated model serving over mcp

Native Node.js bindings for [LiteRT-LM](https://github.com/google/litert-lm) with an API matching the `mcp-llm` Rust crate from [assist-mcp](~/assist-mcp).

## Features

- **Native Performance**: Direct bindings to LiteRT-LM C++ library via node-gyp
- **TypeScript Support**: Full TypeScript types and IDE autocomplete
- **mcp-llm Compatible API**: Same interface as the Rust `mcp-llm` crate
- **Conversation Management**: Multi-turn conversations with context
- **Benchmarking**: Built-in performance metrics (tokens/sec, latency)
- **CPU & GPU**: Support for both CPU and GPU backends

## Prerequisites

1. **Prepare LiteRT-LM runtime artifacts**  
   This repository expects prebuilt LiteRT-LM headers and shared libraries inside `prebuilt/<platform>/`.

   To build them yourself from the LiteRT-LM sources, run:

   ```bash
   cd ~/LiteRT-LM
   bazel build //rust_api:litert_lm_rust_api \
     --define=litert_link_capi_so=true \
     --define=resolve_symbols_in_exec=false
   ```

   Copy the resulting headers and shared libraries into this repo:

   ```bash
   # Headers
   mkdir -p /Users/rpm/gemma3-mcp/prebuilt/include
   cp rust_api/litert_lm_c_api.h /Users/rpm/gemma3-mcp/prebuilt/include/

   # macOS example (adjust for linux/windows)
   mkdir -p /Users/rpm/gemma3-mcp/prebuilt/darwin/lib
   cp bazel-bin/rust_api/*.dylib /Users/rpm/gemma3-mcp/prebuilt/darwin/lib/
   ```

   The build system will automatically copy these `.dylib/.so/

Team