TokShow
a web playground combining a bunch of different LLM tokenizers
Core trick
Runs input text through multiple LLM tokenizer libraries client-side and renders token boundaries side by side.
a web playground combining a bunch of different LLM tokenizers
Runs input text through multiple LLM tokenizer libraries client-side and renders token boundaries side by side.