Shub Tiwari

Toolstoken counter

How A Language Model Reads

Language models don't read letters or whole words. They read tokens: common chunks of text. Type anything in the box below; each coloured block is one token.

~/tools/token-counter
1 · your texto200k_base
2 · what the model seeshover a block for its ID
Tokens
…
Characters
0
Words
0

Uses GPT-4o's o200k_base tokenizer (Claude's counts differ a little). In English, one token is about ¾ of a word. Everything runs in the browser; nothing typed here leaves the page.