What Are Tokens in AI? The 4-Character Rule, Explained

A token in AI is a small chunk of text (a whole word, part of a word, a character, or a piece of punctuation) that an AI model reads and generates. Large language models like ChatGPT, Claude, and Gemini do not actually see “words.” They break text into tokens first, then process those. Rule of thumb in English: 1 token ≈ 4 characters ≈ 0.75 of a word, so 1,000 tokens is around 750 words. Tokens matter because they decide the context window (how much text the model can handle at once) and the cost (AI APIs charge per token). Everything else is decoration.

What a token actually is

When you type a prompt, the AI does not process it letter by letter or as neat dictionary words. It splits your text into tokens, units that can be a full word (“cat”), part of a word (“token” + “ization”), a single character, a space, or punctuation, then works with those. Each token maps to an ID number in the model’s vocabulary, and the model does all its math on those numbers. When it responds, it generates tokens one at a time and converts them back into readable text. You never see the tokens. They are the hidden currency the model thinks in.

True across every modern LLM. ChatGPT, Claude, Gemini, Llama, DeepSeek. All tokens, not words.

How tokenization works

The process of splitting text into tokens is called tokenization. Most modern models use subword tokenization (commonly Byte Pair Encoding, or BPE; see NVIDIA’s explainer on AI tokens). Instead of giving every possible word its own token (which would need a gigantic vocabulary), the model breaks rarer or longer words into smaller, reusable pieces.

Concrete examples:

• “cat” → 1 token. Common short word.
• “tokenization” → often 2 tokens, like `token` + `ization`.
• “ChatGPT” → may split into `Chat` + `GPT` (2 tokens).
• A space or punctuation mark → frequently its own token. The space before a word usually attaches to it.

This is why token counts feel unpredictable: common words are single tokens, unusual words, names, code, or non-English characters get chopped into several. The model learned this vocabulary from huge amounts of text so it can represent almost anything efficiently.

Tokens vs words: the 4-character rule

According to OpenAI’s own guidance, one token is approximately four characters, or about 75% of a word in English.

Text amount	Token count
1 token	≈ 4 characters ≈ ¾ of a word
100 tokens	≈ 75 words (about a paragraph)
1,000 tokens	≈ 750 words
1 page (~500 words)	≈ 660 tokens
1 book (~90K words)	≈ 120K tokens

Approximations, not exact math. Short common words may be one token. Long or rare words take more. Numbers, emojis, code, and languages other than English usually consume more tokens per word.

Why tokens matter, three reasons

Context window. Every model has a maximum number of tokens it can consider in a single conversation. GPT-4 Turbo: 128K tokens (~96K words). Claude 3.5 Sonnet: 200K tokens (~150K words). Gemini 1.5 Pro: up to 2M tokens (~1.5M words). Go over the limit and the model truncates or forgets the earliest text. Understanding tokens is how you reason about “how much can I paste into this prompt?”

Cost. AI APIs from OpenAI, Anthropic, and Google are priced per token, usually with separate rates for input and output tokens (your prompt versus the model’s response). Output tokens often cost more. A wordy prompt and a long answer both cost more, and counting tokens is how developers estimate and control their AI bills. Core knowledge for anyone learning how to become an AI engineer.

Performance. Because models predict the next token based on previous tokens, token boundaries affect behaviour. Awkward tokenization (common with code, rare words, or other languages) can hurt quality and waste context. Efficient prompts use fewer tokens to say the same thing.

Tokens by language and content type

Not all text tokenises equally. Some patterns worth knowing:

Content type	Rough tokens per word (English baseline = 1.33)
Standard English prose	1.3
Technical English / code	1.5-2.0
Chinese / Japanese	1.5-2.5 per character
Non-Latin scripts (Arabic, Hindi, Thai)	2-4 per word
Emoji-heavy chat	3-6 per emoji
URLs	5-10 per URL (rare tokens)

Practical consequence: the same message costs more in Arabic than in English on token-priced APIs. This is a known bias in tokenisation that model providers are gradually improving with better tokenisers, but the gap persists.

Input tokens vs output tokens

APIs distinguish two flavours:

• Input tokens. The prompt you send. Includes system prompts, conversation history, any documents you paste in.
• Output tokens. The text the model generates in response.

Pricing (all approximate as of 2026):

Model	Input ($/1M tokens)	Output ($/1M tokens)
GPT-4o	$2.50	$10.00
GPT-4o mini	$0.15	$0.60
Claude 3.5 Sonnet	$3.00	$15.00
Claude 3.5 Haiku	$0.80	$4.00
Gemini 1.5 Pro	$3.50	$10.50
Gemini 1.5 Flash	$0.075	$0.30

Output is 4-5x more expensive than input on frontier models. That is why “make the model output less” is often the biggest cost lever.

Counting tokens before you send them

Three practical options:

OpenAI’s tokeniser demo. Free web tool at platform.openai.com/tokenizer. Paste text, see token count and the token boundaries highlighted. Useful for building intuition.

Local tokeniser libraries. `tiktoken` for OpenAI models, `anthropic` SDK for Claude tokenisation, `google-generativeai` for Gemini. All available as pip installs. Standard for production apps that need to estimate cost or truncate before hitting the model.

LangChain, LlamaIndex, or SDK-provided count helpers. If you are already building on top of a framework, use the built-in helper. Do not reimplement the tokeniser.

Rough estimation without a library: character count ÷ 4 gets you within 10% for English text. Good enough for napkin math.

Common questions about tokens

Are tokens the same across all models? No. Different models use different vocabularies. The same text tokenises to a slightly different count depending on the model. GPT and Claude use similar (but not identical) BPE approaches. Gemini uses its own tokeniser. Llama and DeepSeek have their own.

Do tokens include spaces? Yes, in most modern tokenisers. The leading space usually attaches to the following word as part of the same token.

Does capitalisation matter? Yes. “Cat” and “cat” may tokenise differently, and “CAT” almost certainly does.

Is there a way to reduce tokens? Yes. Shorter prompts. Removing repeated context. Compressing conversation history. Using cheaper “mini” or “haiku” models for tasks that do not need frontier capability. Structured output formats (JSON) that skip prose padding.

What happens if I exceed the context window? Depending on the provider, either the API returns an error, or the middleware silently truncates the earliest tokens. Neither is ideal. Count before you send.

For the mechanics behind how models actually use those tokens, how AI search engines work covers retrieval and how AI creates images and videos covers the visual side. For the broader vocabulary the token concept sits inside, the AI glossary has plain-English definitions for 264 terms.

The single sentence to remember: models read and write in tokens, and 1 token ≈ 4 characters ≈ 0.75 of a word in English. Everything else is a variation on that ratio.