How many tokens is my prompt? GPT, Claude and Gemini counted on the same text

Every large-language-model API bills by the token, rate-limits by the token, and rejects prompts by the token — yet most people budgeting one have never counted a single token. They divide characters by four and move on. For plain English that’s within 10%. For the things developers actually send — JSON, code, logs, tables, non-English text — it can be off by a factor of two, and the error compounds across every request in production.

This guide covers what a token actually is, how the three major providers differ, and how to get a number you can trust before you commit a budget.

What a token is, concretely

A tokenizer is a lookup table of “pieces” learned from training text. Common English words — including their leading space — are single pieces. Less common words are split into fragments that the model has seen often enough to know. Anything rare dissolves into two- or three-character chunks.

Paste this into the AI Token Counter and turn on Show token boundaries:

The unbelievable tokenizer handles antidisestablishmentarianism poorly.

You’ll see The, tokenizer, handles, poorly as one token each; unbelievable as two or three; and the long word as five or six fragments. That’s the whole mechanism. Three consequences follow from it.

Whitespace and punctuation count. A newline is a token. A run of spaces is a token. Each {, ", : in JSON is a token. A pretty-printed JSON object can be 30% more tokens than the same object minified — the indentation is billed.

Numbers are chunked. OpenAI’s o200k splits digits into groups of up to three, so a 10-digit phone number is four tokens and a 16-digit card number is six. A table of numbers is expensive per byte.

Vocabularies are English-centric. The pieces that exist as single tokens are overwhelmingly English words. Other Latin-script languages do reasonably (French and German run perhaps 20–40% more tokens than English for the same meaning). Non-Latin scripts — Hindi, Arabic, Japanese, Thai — often tokenize close to one token per character, which can be 2–4× the English count. If your product is multilingual, budget per language.

How the three providers differ

	OpenAI (GPT-4o, GPT-5.x, GPT-6)	Anthropic (Claude)	Google (Gemini)
Tokenizer	o200k_base — public, ~200K vocabulary	Proprietary, not published	SentencePiece-based, not published as a library
Exact count in browser?	Yes (open-source reimplementations match the API)	No — estimate only	No — estimate only
Exact count via API?	Yes (usage field on every response)	Yes — free count_tokens endpoint	Yes — free countTokens method
Typical English density	~0.75 words/token	Slightly finer — about 10–15% more tokens than o200k	Close to o200k
Stated rule of thumb	~4 characters/token	~3.5 characters/token	~4 characters/token

The practical upshot: a GPT count from a good browser tool is the number the API will bill. A Claude or Gemini count from any browser tool is an estimate, and honest tools say so. The Claude Token Counter and Gemini Token Counter here run OpenAI’s tokenizer and apply a calibration factor, marking every number with a tilde; when you need reconciliation-grade accuracy, use the providers’ free counting endpoints.

Why “cheaper per token” can bill more

Suppose Model A charges $2 per million input tokens and Model B charges $1.80. B looks 10% cheaper. But if B’s tokenizer produces 15% more tokens on your text, B bills more. The only fair comparison is tokens on your text × price, per model — which is what the cost table under each counter does. Per-token price lists, on their own, aren’t comparable across providers.

The same logic applies within a provider over time. When OpenAI moved from cl100k_base (GPT-4) to o200k_base (GPT-4o), the same English text got roughly 5–15% fewer tokens, so effective prices dropped even where list prices didn’t.

Counting the parts of a real request

Your bill isn’t just the user’s message. A production request typically has:

1. System prompt — often the largest single component, and repeated on every call. A 1,500-token system prompt at 10,000 requests a day is 15 million input tokens a day before a user types anything. This is what prompt caching (offered by OpenAI and Anthropic at roughly 10% of the input price for the cached prefix) is for.
2. Tool and function definitions — JSON schemas, sent with each request that can call tools. Frequently 500–3,000 tokens and frequently forgotten in estimates.
3. Conversation history — grows every turn. A 20-turn chat re-sends all 19 previous turns on turn 20.
4. Retrieved context — documents pulled in by RAG. Usually the most variable component.
5. The user’s message — usually the smallest.
6. Output — billed at 5–6× the input rate on most current models, so a 400-token answer often costs more than a 1,500-token prompt.

Count each separately once, then you can reason about which one to shrink. In practice the wins are, in order: cache the system prompt, trim tool schemas to the tools the request actually needs, summarise history past a window, and cap output length in the instruction.

Getting an exact number without uploading anything

Most token-counter sites send your text to a server to count it. That’s an odd thing to accept for the one category of text — system prompts, proprietary context, customer data — that you least want to leak. The counter linked from this guide runs the o200k_base tokenizer as JavaScript in your browser tab: the ~2 MB vocabulary downloads once, your text goes nowhere, and you can verify that by loading the page and switching off your network before pasting.

For a quick calibration of your own intuition, paste three things and compare: a paragraph of prose, the same paragraph as a JSON string with escaped quotes, and a 30-line function. You’ll never trust the four-characters rule again.

Checklist

• Count the whole request — system prompt, tools, history, context, message — not just the message.
• Compare models on tokens × price for your text, never on list price alone.
• Treat Claude and Gemini browser counts as estimates; use the free API endpoints when exactness matters.
• Minify JSON, strip comments, shorten keys — the cheapest token savings there are.
• Budget per language if you serve non-English users.
• Remember output is billed at 5–6× input; a shorter answer often saves more than a cheaper model.