CrossEntropyLoss ignore_index removes supervision at positions whose target has the ignored value. It does not delete a class from every softmax. Padding targets can be excluded from the loss while valid positions still normalize across all output classes.

This explains why a padding-class logit can affect loss even after padding labels are ignored. A three-class zero-logit example gives log 3; actually removing one class gives log 2. The difference is a change of problem, not a numerical inconsistency. 中文版.

Label the position axis and the class axis

The official integer-target definition uses an indicator for whether a position contributes. Its softmax denominator still sums over all C classes. This derivation assumes hard labels, no class weights and no label smoothing, keeping distinct reduction rules out of the example.

In sequence labeling, one axis identifies a token and another lists possible classes for that token. Ignoring a token’s target and prohibiting other tokens from predicting a class operate along different axes. Calling both things padding makes their separate intentions easy to merge accidentally.

An ignored value such as −100 can sit outside the class range. It is a target-side sentinel, not an output class numbered minus one hundred. The input still supplies logits for the actual classes. Do not add a meaningless output column or pass the sentinel through ordinary class-indexing logic.

Three zero logits provide a minimal counterexample

Take logits [0,0,0] at a valid position with target class one. Every exponential is one, so target probability is one-third and loss is log 3. Giving another position an ignored target only removes supervision at that other position.

Actually deleting class zero, and remapping original target one to index zero in the remaining space, produces logits [0,0]. Only then does target probability become one-half and loss log 2. The first calculation asks for the target probability among three classes; the second changes the class space. An ignored target does not express both operations.

Now increase class zero’s logit from zero to five at the valid position. Even if that class is never the correct label for this position, it remains in the denominator, reducing target probability and raising loss. Whether that competition is wanted is a modeling decision that ignore_index does not make.

The gradient also reveals class competition

For one nonignored, unweighted position, the derivative with respect to logit zₖ is pₖ−1[k=y]. A nontarget class still in the normalization set has a gradient whenever its probability is nonzero. In the three-class zero-logit example, each nontarget derivative is one-third.

This does not assign the same direct supervision to an ignored row. Ignoring its target removes that row’s direct contribution to this loss. Earlier shared parameters or attention paths create a separate question: a token that is not a supervised target need not disappear from every path in the full model calculation.

The earlier padding_idx and attention-mask counterexample examines input participation. This article examines output supervision and class normalization. A complete padding contract normally defines those responsibilities separately.

Nine checks distinguish the two operations

The teaching function represents an unsupervised row using None and explicitly rejects a batch with no supervised positions. That rejection is an application choice, not a claim about every framework’s edge-case return value. Subtracting the maximum logit avoids directly evaluating exp(1000).

# Hard-label, unweighted, unsmoothed teaching loss; no PyTorch execution.
from math import exp, log, isclose

def ce(logits, target, ignore=-100):
    if target == ignore:
        return None
    if not 0 <= target < len(logits):
        raise ValueError("invalid target")
    m=max(logits)
    return log(sum(exp(z-m) for z in logits))+m-logits[target]

def valid_mean(logits, targets):
    if len(logits)!=len(targets):
        raise ValueError("position count mismatch")
    losses=[ce(row,y) for row,y in zip(logits,targets)]
    kept=[x for x in losses if x is not None]
    if not kept:
        raise ValueError("no supervised positions")
    return sum(kept)/len(kept)

assert isclose(ce([0.,0.,0.],1),log(3))
assert isclose(ce([0.,0.],0),log(2))
assert ce([99.,0.,0.],-100) is None
assert ce([5.,0.,0.],1)>ce([0.,0.,0.],1)
assert isclose(valid_mean([[0.,0.,0.]]*2,[1,-100]),log(3))
assert isclose(valid_mean([[0.,0.,0.]]*3,[1,-100,-100]),log(3))
try:
    valid_mean([[0.,0.,0.]],[-100])
except ValueError:
    pass
else:
    raise AssertionError("expected empty supervision rejection")
assert isclose(ce([1000.,1000.,1000.],1),log(3))
try:
    valid_mean([[0.,0.,0.]],[])
except ValueError:
    pass
else:
    raise AssertionError("expected shape rejection")
print("9 ignore-index arithmetic checks passed")

All nine assertions passed locally. They compare log 3 with log 2, ignore a complete row, increase loss by raising a competing logit, and preserve the valid-position mean when ignored positions are added. They also reject all-ignored input and mismatched position counts, and check a common logit shift. The code reproduces this particular mathematical definition without running PyTorch training.

Match the mean denominator to supervision

In the unweighted setting here, sum losses over nonignored positions and divide by their count. Replacing ignored losses with zero and averaging over every padded position instead makes added padding reduce the mean. That decline does not establish better predictions on valid tokens.

With class weights, the integer-target mean involves the sum of weights for valid target classes, so a simple valid count is insufficient. Probability targets are another API mode, and ignore_index does not apply to them. Changing label representation, weighting or smoothing requires checking the corresponding formula again.

An all-ignored batch provides no supervision denominator for this mean. An application can skip the training unit or define another explicit policy, but that policy must agree with accumulation, distributed reduction and metric aggregation. Replacing an exceptional result with zero while treating it as an ordinary update can obscure what a training step means.

Define the new problem before excluding a class

Permanently removing an output class requires consistent output columns, target remapping and inference semantics. Deleting a column without adjusting later indices can leave a computable loss aimed at the wrong labels. A class forbidden only at selected positions instead requires a position-dependent allowed set.

A row with no allowed class needs a separate policy. An arbitrary finite negative logit is not proof of semantic deletion: its numerical effect depends on dtype, range and subsequent operations. First decide whether a real class constraint exists, then choose a representation and tests that express it.

A language model’s pad symbol may share an ID with another purpose or be constrained separately during decoding. Removing it from training normalization must follow that model’s contract. A smaller loss in one toy example is not a reason to change pretrained vocabulary semantics.

Name tests after the axes they check

Prepare one valid target and one ignored target, examining unreduced losses. Change only a nontarget logit in the valid row, then add only ignored positions. These checks separately test position filtering, class competition and the reduction denominator. A failure identifies a different operation in each case.

Record the class axis explicitly, particularly when a sequence tensor changes from [N,T,C] to another layout. Inspecting the loss shape and position correspondence reveals more than a final scalar. Keep the target mapping beside the example so correct-looking numbers do not hide incorrect label meaning.

If the derivation and reproducible checks help distinguish ignoring labels from deleting classes, voluntary support through the section below is welcome. Keeping the counterexample as a regression test makes future label or vocabulary changes easier to verify.

Sources and backfill note

CrossEntropyLoss

The definitions were checked against PyTorch 2.8 material available before the planned slot. Derivations, counterexamples and figures are original teaching material. The article keeps October 19, 2026 at 14:00 Beijing time as its planned date and was backfilled that evening. Standard-library code ran locally; no PyTorch model or device-performance benchmark ran.