OpenAI Tiktoken & LLM Token Counter

The Mechanics of Tokenization: Byte-Pair Encoding (BPE), Vocabulary Sizing & API Economics

In the architecture of natural language processing (NLP) and modern neural networks, a fundamental axiom states that computers process numbers, not words. Before a Large Language Model (such as OpenAI GPT-4o, Anthropic Claude 3.7 Sonnet, Google Gemini 2.0, or DeepSeek R1) can apply matrix multiplications or evaluate attention heads, the raw UTF-8 string input must be translated into an ordered sequence of discrete numerical identifiers known as tokens.

Tokens do not map one-to-one with English words or letters. Instead, modern foundation models employ statistical sub-word algorithms—most notably Byte-Pair Encoding (BPE) and SentencePiece. Understanding how these algorithms partition text, how vocabulary size impacts non-English tokenization, and how token counts dictate context windows and cloud API billing is an essential discipline for AI software engineers and prompt architects.

LLM Tokenizer Encodings, Vocabulary Sizes & API Economics Reference Table

The comparative reference table below provides a comprehensive breakdown of the major tokenization algorithms, vocabulary volumes, compression ratios, and pricing economics across frontier LLMs:

How Byte-Pair Encoding (BPE) Works Under the Hood

The Byte-Pair Encoding algorithm was originally developed in 1994 as a general-purpose data compression technique (Gage, 1994) before being adapted for neural machine translation by Sennrich et al. in 2016. In an LLM tokenizer:

1. Base Byte Vocabulary: The tokenizer starts with a base alphabet of 256 byte values (representing all possible single bytes in the raw UTF-8 standard). This guarantees that the tokenizer can process any arbitrary string or binary character without throwing "out-of-vocabulary" (OOV) errors.
2. Statistical Frequency Merging: During tokenizer training across terabytes of multilingual text and source code, the algorithm counts the most frequently co-occurring adjacent pairs of bytes or characters.
3. Iterative Token Creation: The highest-frequency pair (e.g., the bytes for 't' and 'h') is merged into a new, composite token ('th'). This merge process repeats tens of thousands of times until the vocabulary reaches its target size (e.g., 100,000 in cl100k_base or 200,000 in o200k_base).

Common English words like "the", "example", and "algorithm" become single tokens. Infrequent, compound, or specialized technical words are split into several sub-word tokens. For example, "cryptocurrency" is split into ["crypto", "currency"].

The Multilingual Token Inflation Penalty

Because early tokenizers (such as GPT-2's 50k vocabulary and GPT-4's 100k vocabulary) were trained primarily on English-dominant web scrapes, non-Latin alphabets—such as Cyrillic, Arabic, Devanagari (Hindi), Chinese, and Japanese—suffered from severe token inflation.

In UTF-8 encoding, an ASCII English letter requires exactly 1 byte. In contrast, a Hindi or Chinese character requires 3 or 4 bytes. In smaller vocabularies lacking pre-merged non-English words, each non-Latin character could consume 2 to 4 separate tokens. Consequently, translating an English paragraph into Hindi or Arabic could triple the prompt's token count and triple API costs for identical semantic information.

OpenAI's introduction of `o200k_base` in GPT-4o doubled the vocabulary size to 200,000 tokens. This dramatic expansion allowed thousands of common non-English words and multilingual phrases to be assigned dedicated single-token IDs, reducing non-English token consumption by 20% to 50% across major world languages.

The Economics of Prompt Caching & Context Windows

When engineering agentic loops, multi-turn chat applications, or RAG (Retrieval-Augmented Generation) pipelines, token counting is directly coupled to API latency and unit economics:

• Input vs. Output Token Asymmetry: Output tokens are universally priced 3x to 5x higher than input tokens. This disparity exists because input tokens can be processed in parallel across GPU matrix cores, whereas output tokens must be generated auto-regressively, one token at a time, requiring sequential key-value (KV) cache memory access.
• Prompt Caching Optimization: Frontier providers like Anthropic, OpenAI, and DeepSeek offer Prompt Caching. If the initial prefix of your prompt (e.g., large system instructions, documentation context, or schema definitions) exceeds a threshold (typically 1,024 tokens) and remains static across requests, subsequent calls receive up to a 50% to 90% discount on those cached input tokens.