"What Is a Token in AI? The 2026 Plain-English Guide"
A token is the basic unit of text that an AI model processes. It is roughly a word, a part of a word, or a character — depending on the language and tokenizer. Here is everything you need to know.
The simple version
- 1 token ≈ 4 characters of English text
- 1 token ≈ 0.75 words of English
- 100 tokens ≈ 75 words
So a 1,000-word blog post is about 1,333 tokens. A 50,000-word book is about 66,667 tokens.
How many tokens common things use
| Input | Approx. tokens |
|---|---|
| 1 word (English) | ~1.3 |
| 1 word (Chinese) | ~2–3 |
| 1 emoji | ~2–4 |
| 1 line of Python code | ~10–15 |
| A tweet (280 chars) | ~70 |
| A news article (~500 words) | ~667 |
| A blog post (~1,500 words) | ~2,000 |
| A chapter of a book (~5,000 words) | ~6,667 |
| A full book (~80,000 words) | ~106,667 |
Why tokens matter for cost
API pricing is per million tokens. Here is what common tasks cost on GPT-4o ($2.50/$10 per M tokens):
| Task | Input tokens | Output tokens | Cost |
|---|---|---|---|
| Single chat message | ~50 | ~200 | $0.002 |
| Blog post generation | ~100 | ~1,500 | $0.015 |
| Document summarization (10 pages) | ~3,000 | ~500 | $0.013 |
| Code review (100-line file) | ~300 | ~300 | $0.003 |
| 1,000 chat conversations | ~50K | ~200K | $2.13 |
Context window limits
Each model has a maximum context window — the total tokens it can process in one call:
| Model | Context window | Approx. English words |
|---|---|---|
| GPT-4o | 128K | ~96,000 |
| Claude 3.5 Sonnet | 200K | ~150,000 |
| Gemini 1.5 Pro | 2M | ~1,500,000 |
| GPT-4o mini | 128K | ~96,000 |
| Llama 4 70B | 128K | ~96,000 |
How to count tokens
OpenAI models: Use the tiktoken Python library:
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Your text here")
print(len(tokens))
Anthropic models: Use their tokenizer or estimate at ~1.3 tokens per word.
Online: OpenAI’s tokenizer page (platform.openai.com/tokenizer) lets you paste text and see tokens.
FAQ
Why not just count words? Because tokens are not words. “Strawberry” is 2 tokens. “🚀” is 2–3 tokens. Tokenization is optimized for compression, not for human word boundaries.
How many tokens is a system prompt? A typical system prompt (“You are a helpful assistant…”) is 50–200 tokens. A complex system prompt with instructions, examples, and rules can be 1,000+ tokens — and you pay for it on every API call (unless you use prompt caching).
Do images count as tokens? Yes. On GPT-4o, a standard image is ~1,000–2,000 tokens depending on resolution.
Verdict
Understanding tokens is essential for estimating API costs and managing context windows. Rule of thumb: 1 word ≈ 1.3 tokens (English), and always use a tokenizer for precise counts. Every token costs money — so write efficiently, cache system prompts, and use the cheapest model that meets your quality bar.