Cross-entropy is the average cost of describing a target distribution p with predictions q. KL divergence is the part above p’s own entropy: H(p,q)=H(p)+KL(p‖q). For fixed p, optimizing either has the same objective, but the numbers mean different things.

Calculate a two-event example

Take p=[0.9,0.1] and q=[0.5,0.5], with corresponding positions representing the same events. Using natural logarithms gives entropy about 0.325083, cross-entropy 0.693147 and KL 0.368064, all in nats. Thus 0.693147≈0.325083+0.368064.

These are constructed probabilities, not model or dataset measurements. Because q is uniform, both events have negative log probability log 2. The target p favors the first event and has lower intrinsic uncertainty.

The difference is KL divergence

For a shared discrete event space, H(p)=-Σᵢpᵢlog pᵢ and H(p,q)=-Σᵢpᵢlog qᵢ. Subtracting gives Σᵢpᵢ(log pᵢ-log qᵢ), which is KL(p‖q). Terms with pᵢ=0 contribute zero. If pᵢ>0 while qᵢ=0, both KL and cross-entropy are infinite.

With p fixed, H(p) does not change with q, so minimizing cross-entropy and this direction of KL has the same optimum. If p changes too, for example when both distributions depend on trainable parameters, its entropy cannot simply be discarded as a constant.

The discussion concerns discrete distributions. The official SciPy 1.15.2 entropy documentation gives these definitions. That function normalizes unnormalized inputs; the teaching code below rejects them instead so that it does not silently change the input.

Reversing the direction changes the question

KL(p‖q) weights terms by p; KL(q‖p) weights them by q. The reversed example gives about 0.510826 rather than 0.368064. It is not an ordinary distance whose arguments can be swapped freely.

If an event has high target probability but low predicted probability, forward KL weights that cost by the target. Reversal changes the weights too. When comparing teacher and student distributions, state which distribution supplies the expectation rather than merely adding an unspecified KL loss.

Event IDs must also align. If the first position means token A in one distribution and token B in the other, equal shapes do not make the comparison meaningful. Different tokenizer vocabularies cannot be compared position by position without a valid alignment, a consequence of the binding described in BPE tokenization.

Check the identity and zero probabilities in Python

import math

def validate(p, q):
    if not p or len(p) != len(q):
        raise ValueError("Expected equal nonempty distributions")
    for x in (p, q):
        if not all(math.isfinite(v) and v >= 0 for v in x):
            raise ValueError("Invalid probability")
        if not math.isclose(math.fsum(x), 1.0):
            raise ValueError("Probabilities must sum to one")

def metrics(p, q):
    validate(p, q)
    entropy = -math.fsum(a * math.log(a) for a in p if a > 0)
    if any(a > 0 and b == 0 for a, b in zip(p, q)):
        return entropy, math.inf, math.inf
    cross_entropy = -math.fsum(a * math.log(b)
                               for a, b in zip(p, q) if a > 0)
    kl = math.fsum(a * (math.log(a) - math.log(b))
                   for a, b in zip(p, q) if a > 0)
    return entropy, cross_entropy, kl

p, q = [0.9, 0.1], [0.5, 0.5]
h, ce, kl = metrics(p, q)
assert math.isclose(ce, h + kl)
assert math.isclose(metrics(p, p)[2], 0.0, abs_tol=1e-12)
assert not math.isclose(kl, metrics(q, p)[2])
assert math.isinf(metrics([1., 0.], [0., 1.])[2])
assert metrics([1., 0.], [1., 0.]) == (0., 0., 0.)
for bad in ([0.2, 0.2], [-0.1, 1.1], [math.nan, 0.]):
    try:
        metrics(bad, q)
    except ValueError:
        pass
    else:
        raise AssertionError("Invalid input accepted")
print([round(x, 6) for x in (h, ce, kl)])
print(round(metrics(q, p)[2], 6))

This original code was executed, checking the decomposition, zero KL for matching distributions, asymmetry, infinity when q assigns zero to positive target mass, and invalid inputs. The first three values are [0.325083,0.693147,0.368064]; reverse KL is 0.510826.

The code uses natural logarithms. For n events it takes O(n) time and O(1) extra space excluding inputs, without constructing a work list growing with n. Floating-point rounding remains; tiny numerical deviations do not refute the identity.

A hard label and a soft distribution differ

For a one-hot target with probability 1 on class y, H(p)=0, and cross-entropy and KL both become -log qᵧ. A soft target generally has nonzero entropy, so its cross-entropy need not become zero when predictions match it.

Loss values across different targets, label smoothing or reductions therefore cannot be compared casually. Record target definitions, probability versus logit inputs, logarithm base, reduction dimensions and zero-probability handling before interpreting values. No training-effect experiment was run, and this derivation does not establish that one KL direction always produces a better model.

阅读中文版

Sources

• docs.scipy.org — original paper / versioned documentation

Backfilled on October 13, 2026. The article date retains its planned Beijing slot. References predate that slot; the examples are original teaching calculations.