A model can receive different perplexity scores when the evaluation protocol changes. PPL is not answer accuracy: it measures the conditional probabilities assigned to observed target tokens. The denominator, probabilities and context must be clear before scores are comparable.

Compute PPL from three probabilities

Let N be the number of scored tokens and p₁ through pₙ their conditional probabilities. With natural logarithms, mean negative log-likelihood is L=−Σln(pᵢ)/N and perplexity is exp(L). Use the probability of each observed target token, not the largest probability among candidates.

In a fictional example, observed targets receive probabilities 1/2, 1/4 and 1/8. Their product is 1/64, so the geometric mean probability is 1/4 and PPL is 4. With base-two logarithms, the surprises are 1, 2 and 3 bits. Their mean is 2 bits, and 2² again gives 4. Do not insert a base-two average loss into exp.

Why the arithmetic probability mean differs

The arithmetic mean of these probabilities is 7/24. Its reciprocal is about 3.428571, not PPL=4. PPL depends on average log probability; a low-probability observed target increases negative log-likelihood. Averaging probabilities first calculates a different statistic.

Assigning probability 1/4 to every observed target also gives PPL=4. Equal perplexity can therefore come from different probability patterns. It does not locate individual answer errors or establish that exactly four candidates exist at every step.

Weight pooled text by scored token counts

Consider two fictional texts: a short text has one scored token with probability 1/2 and PPL=2; a longer text has three scored tokens, each with probability 1/8, and PPL=8. Averaging the two PPL values gives 5. Pooled corpus PPL is instead √32≈5.656854.

Pool negative log-likelihood sums, divide by the total number of scored tokens, then exponentiate. A mean of per-document PPL values can be a separate deliberate metric, but label it clearly rather than calling it corpus perplexity.

import math

def ppl(probs):
    if not probs or any(not 0 < p <= 1 for p in probs):
        raise ValueError("Expected nonempty probabilities in (0, 1]")
    return math.exp(-sum(math.log(p) for p in probs)/len(probs))

assert math.isclose(ppl([0.5,0.25,0.125]),4.0)
assert math.isclose(ppl([1.0,1.0]),1.0)
assert math.isclose(ppl([0.25]*3),4.0)
short, long = [0.5], [0.125]*3
pooled = ppl(short+long)
assert math.isclose(pooled, math.sqrt(32))
assert not math.isclose(pooled,(ppl(short)+ppl(long))/2)
for bad in [[],[0.0],[1.1],[-0.1]]:
    try:
        ppl(bad)
    except ValueError:
        pass
    else:
        raise AssertionError("Invalid input accepted")
print(round(pooled,6))

This original Python example was executed, checking the three-probability calculation, the PPL=1 boundary, unequal-length pooling and invalid inputs. It runs no language model and reports no real model score. Zero probability theoretically gives infinite loss; this example rejects it. Real evaluation generally works directly with log probabilities to avoid underflow from converting to probabilities first.

Tokenization and context are evaluation conditions

The versioned Hugging Face documentation notes that tokenization and context windows affect PPL evaluation. Record tokenizer, dataset, scored targets, context policy and model version. A per-token unit is not identical across tokenization schemes.

Disjoint chunks discard preceding context at chunk boundaries. Sliding windows retain more context, but overlapping tokens used as context should not be scored again. Specify handling of first positions, padding and unscored prompt text. This article does not implement framework-specific label shifting; the denominator must count targets actually used in the loss.

What PPL can tell you

Under a consistent protocol, it measures probability modeling on target text. It does not directly check facts, citations, instruction following or code correctness. See pass@k for code evaluation and RAG evaluation for retrieval-answer failure diagnosis.

Likelihood evaluation using original model probabilities differs from randomly sampling after temperature or top-p adjustments. Sampling settings are not the definition of the model’s original likelihood.

中文版

Source

Hugging Face Transformers v4.50.0: Perplexity of fixed-length models