"What Is Tokenization? How AI Models Read Text"
Before an AI model can process text, it must break it into tokens. Tokenization sounds trivial — but it affects cost, performance, and even model behavior.
What is a token?
A token is a chunk of text — roughly a word, a part of a word, or a character. The model does not see characters or words; it sees token IDs (numbers).
Examples (GPT-4 tokenizer): - “Hello” → 1 token (9906) - “Hello!” → 2 tokens (9906, 0) - “Strawberry” → 2 tokens (5654, 16147) — “Straw” + “berry” - “unhappiness” → 3 tokens — “un” + “happiness” → “un” + “happi” + “ness” - “🦋” → 3–4 tokens (emoji are expensive)
Why tokenization matters
1. Cost
API pricing is per-token. A document with 10,000 words costs differently depending on the tokenizer: - English text: ~0.75 tokens per word - Chinese text: ~1.5–2 tokens per character (Chinese is token-expensive) - Code: varies — Python is efficient, Rust is less so
2. Model behavior
The famous “how many R’s in strawberry” question trips up LLMs because “strawberry” is tokenized as [“straw”, “berry”] — the model does not “see” individual letters. It sees two tokens, and counting letters within a token is not something transformers do naturally.
3. Language fairness
English is token-efficient (~0.75 tokens/word). Other languages are not: - Spanish: ~1.0 tokens/word - Hindi: ~2.5 tokens/word - Korean: ~3.0 tokens/character
This means non-English API calls are 2–4× more expensive — a real fairness issue.
How tokenizers work
Most modern LLMs use Byte-Pair Encoding (BPE): 1. Start with all individual characters as tokens 2. Find the most common pair of adjacent tokens 3. Merge them into a new token 4. Repeat until vocabulary reaches the target size (typically 50K–128K tokens)
This produces a vocabulary where common words are single tokens and rare words are split into subword tokens.
Different models, different tokenizers
| Model | Tokenizer | Vocab size | Notes |
|---|---|---|---|
| GPT-4 | tiktoken (BPE) | ~100K | Efficient for English, code |
| Claude 3.5 | Custom BPE | ~100K | Similar efficiency |
| Gemini 1.5 | SentencePiece | ~256K | Better multilingual |
| Llama 4 | Tiktoken-style | ~128K | Good English/code |
| Qwen 3 | Custom | ~150K | Optimized for Chinese |
Practical implications
- Estimate costs: 1,000 words ≈ 750 tokens (English), ~1,500 tokens (Chinese)
- Token limits: GPT-4o’s 128K context means ~96K words of English or ~64K characters of Chinese
- Code: Python is token-efficient; minified JS is not
- Formatting: Markdown headers (
##,**) add tokens; plain text is cheaper
FAQ
Why does ‘strawberry’ have 2 tokens? Because BPE found that “straw” and “berry” appear frequently as separate tokens in training data. The tokenizer optimizes for compression, not for human intuition about words.
Can I choose my own tokenizer? Only if you train your own model. When using GPT-4 or Claude, you use their tokenizer.
How do I count tokens? Use OpenAI’s tiktoken library (tiktoken.count("your text")) or the tokenizer page on platform.openai.com. For other models, use their respective tokenizers.
Verdict
Tokenization is the invisible layer between human text and AI models. It affects cost (non-English is more expensive), behavior (token boundaries affect what the model “sees”), and fairness (English is privileged). Understanding it helps you write more token-efficient prompts and estimate API costs accurately.