StringZilla
Fastest tools for LLMs to query large codebases and bring Gemini to Linux- and LLVM-scale environments
Core trick
Uses StringZilla's SIMD substring search, including AVX-512 and SVE paths, to scan files larger than memory in a grep-style Rust CLI.
Tags
Description
Fastest tools for LLMs to query large codebases and bring Gemini to Linux- and LLVM-scale environments
README excerpt
# StringZilla 🦖 Command-Line Interface
Most text processing command-line utilities have obscure syntax, limited portability across operating systems, can't deal with larger-than-memory datasets, and are not actively leveraging modern SIMD capabilities, such as AVX-512 on x86 and SVE on ARM.
This utility is written in Rust, leveraging StringZilla for both pipe and file-based text processing across Linux, macOS, and Windows.
Install it straight from the GitHub repository:
```bash
cargo install --git https://github.com/ashvardanian/StringZilla-CLI
```
It provides the following subcommands:
- `sz-find`: find all inclusions of a substring in a file similar to `grep`, but with a saner syntax
- `sz-replace`: replace a substring in a file or stream; a literal-match alternative to `sed s///`
- `sz-outline`: provide an LLM with an outline of a Markdown or C/C++ source file
- `sz-count`: 3x faster `wc` word count, that can actually handle UTF-8 properly
- `sz-dedup`: deduplicate lines; safe for larger-than-memory files
- `sz-split`: 4x faster `split` file splitting, that won't break UTF-8 characters or lines
- `sz-cols`: extract columns from delimited text; replaces `cut -f` and `awk '{print $N}'` with simpler syntax
- `sz-rows`: extract rows by index or range; replaces `sed -n`, `head`, `tail`, and `awk 'NR==N'`
- `sz-sort`: sort lines; Unicode-correct and