BPE tokenization learns frequent adjacent-symbol merges rather than storing every possible word. Training learns a merge order; encoding reuses it. Token counts therefore depend on the vocabulary and preprocessing, not a universal conversion from characters or English words.

Follow a single merge

Use a fictional corpus with cat appearing three times and car twice. Start from characters. The weighted count of c+a is 5, a+t is 3, and a+r is 2. The first merge creates ca, leaving [ca,t] and [ca,r].

Each occurrence now uses two symbols instead of three. That is a result for one merge in this tiny example, not a claimed reduction for a real model. A different corpus can produce a different merge priority.

Word frequency belongs in the count

Counting each distinct word only once would give c+a a count of 2, discarding how often it actually occurs. The implementation stores symbol sequences and occurrence counts in a Counter, then weights each adjacent pair by its sequence frequency.

Ties need a fixed rule for reproducibility. This example breaks ties lexicographically by symbol pair. That is an explicit teaching convention, not a claim about the training tools used by a particular model.

Learning rules differs from applying them

Sennrich, Haddow and Birch’s ACL 2016 paper uses subword units to address rare words in machine translation. Its BPE method supplies the background here. This is not a new paper report, and no experimental scores from the authors are reused.

Training needs a corpus to determine merge order. An already trained tokenizer should not relearn merges from each request’s word frequencies: the same text could then become different tokens in different requests. A model’s embeddings and output vocabulary are already tied to particular token IDs.

A real encoder also needs saved merge ranks and specified initial splitting, boundaries, special tokens and unknown-symbol handling. This example implements one character-level training merge. It is neither a complete encoder nor byte-level BPE.

Code: count and merge adjacent pairs

from collections import Counter

def count_pairs(words):
    counts = Counter()
    for symbols, frequency in words.items():
        for pair in zip(symbols, symbols[1:]):
            counts[pair] += frequency
    return counts

def merge_pair(words, pair):
    result = Counter()
    for symbols, frequency in words.items():
        merged, i = [], 0
        while i < len(symbols):
            if tuple(symbols[i:i+2]) == pair:
                merged.append(symbols[i] + symbols[i+1])
                i += 2
            else:
                merged.append(symbols[i])
                i += 1
        result[tuple(merged)] += frequency
    return result

words = Counter({tuple("cat"): 3, tuple("car"): 2})
counts = count_pairs(words)
assert counts[("c", "a")] == 5
assert counts[("a", "t")] == 3
best = min(counts, key=lambda pair: (-counts[pair], pair))
assert best == ("c", "a")
merged = merge_pair(words, best)
assert merged == Counter({("ca", "t"): 3, ("ca", "r"): 2})
assert merge_pair(Counter({tuple("aaa"): 1}), ("a", "a")) == Counter({("aa", "a"): 1})
assert count_pairs(Counter({("x",): 1})) == Counter()
assert count_pairs(Counter()) == Counter()
print(best, dict(merged))

The code was executed. Checks cover frequency weighting, the first merge, left-to-right nonoverlapping merging of aa in aaa, and single-symbol or empty corpora. It prints (‘c’,’a’) and then {(‘ca’,’t’):3,(‘ca’,’r’):2}.

One counting pass and one merge pass scan the current sequences. For L symbols across distinct sequences and P distinct pairs, counting is approximately O(L), selecting a pair O(P), and merging O(L), with extra space for the output. These are costs of the simple implementation; full large-corpus training requires other data structures and engineering optimizations.

Use the actual tokenizer for context budgets

An API’s token budget depends on the final encoding and can also include message framing and special tokens. Neither one Chinese character nor one English word has a universal one-to-one relationship with a token. Estimates help initial screening; exact limits need the tokenizer matching the target model and the interface’s counting rules.

For RAG chunking, assuming character boundaries are token boundaries distorts context budgets. For perplexity comparisons, changing the tokenizer changes the per-token unit.

A larger vocabulary may shorten some sequences, but it also changes embedding and output layers, coverage and training requirements. No model experiments on those tradeoffs were run here. The engineering takeaway is to record tokenization as an explicit condition.

阅读中文版

Sources

• aclanthology.org — original paper / versioned documentation

Backfilled on October 13, 2026. The article date retains its planned Beijing slot. References predate that slot; the examples are original teaching calculations.