With nonzero label_smoothing, the target probability of the correct class generally stops being one, and cross entropy no longer has zero as its ideal endpoint. Under uniform mixing, three classes, smoothing 0.3 and class zero as the label give target (0.8,0.1,0.1), rather than (0.7,0.15,0.15). That difference changes the loss, gradient and optimal logit gap together.

The following objective can be calculated by hand and executed independently: check the distribution rule first, explain why an overconfident prediction can be pushed back, and separate a smoothed training objective from ordinary negative log-likelihood. 中文版.

Check which classes receive the smoothing mass

The PyTorch 2.8 CrossEntropyLoss definition describes label smoothing as mixing the original target with a uniform distribution. This article restricts the analysis to one example, C≥2, no class weights and no ignored entries. With mixture proportion ε, write qⱼ=(1−ε)·1[j=y]+ε/C.

The uniform component includes the correct class, so that class receives ε/C too. With C=3 and ε=0.3, it gets 0.7+0.1=0.8, while each remaining class gets 0.1. A different implementation that distributes all 0.3 only among the two wrong classes produces (0.7,0.15,0.15). Both are valid distributions, but they define different objectives.

When comparing implementations, matching parameter names and ε values is insufficient. Inspect the actual target vector, its sum, the correct-class mass and the distribution across other classes. The endpoints also have concrete meanings: ε=0 restores a hard target; ε=1 produces a fully uniform target with no preference for this example’s original class. It still contributes a loss and is not equivalent to ignoring the example. Changing C changes the per-class allocation at fixed ε. A three-class derivation must therefore be recomputed before applying its numerical values to a large vocabulary.

The method’s formula can be checked in Section 7 of the 2015 Inception preprint, version 3. This is a reference to the public preprint’s definition, not a claim of a recent breakthrough. Image-classification results are not reused as a promise about a different task.

More confidence need not reduce the smoothed loss

For fixed target q, define L=−Σⱼqⱼlog pⱼ, with p obtained by applying Softmax to logits z. Without weights and with Σq=1, rewrite the objective as L=logΣⱼexp zⱼ−Σⱼqⱼzⱼ. Differentiating gives ∂L/∂zⱼ=pⱼ−qⱼ. The direction follows directly from the new target rather than an unexplained penalty added to a conventional loss.

At p=(0.98,0.01,0.01), a hard target gives correct-class derivative 0.98−1=−0.02, so gradient descent still raises that logit. The smoothed target gives 0.98−0.8=0.18, so gradient descent lowers it instead. Each incorrect class has derivative 0.01−0.1=−0.09. The predicted class is still right, but its confidence has passed the level favored by this particular objective.

This is normal behavior after changing the objective, rather than a runtime defect that punishes a correct classification by accident. It explains one gradient direction. It does not establish that trained accuracy, calibration or out-of-distribution behavior will improve. Those questions require separate data and actual experiments.

Compute the nonzero loss floor from target entropy

For positive q, cross entropy decomposes into H(q)+KL(q∥p). Since KL is nonnegative, freely choosing a predictive distribution minimizes the objective at p=q, with minimum H(q). In the three-class example, that floor is approximately 0.639032. A loss that remains above it is not by itself evidence that an optimizer is malfunctioning.

At p=q=(0.8,0.1,0.1), the correct logit can exceed either wrong-class logit by log(0.8/0.1)=log 8, approximately 2.07944. Adding the same constant to every logit leaves probabilities unchanged. A hard target instead places its ideal prediction on the boundary (1,0,0), which finite logits cannot reach exactly.

The floor concerns one freely representable target distribution. A shared-parameter model facing many different inputs may not fit every target exactly. Class weights, additional regularization terms and reductions also change the reported value. Treating 0.639032 as the expected convergence value of every model would be another misuse of the example.

A useful diagnostic is the current loss minus the entropy of that same fixed target. Under the stated conditions, the difference is precisely KL. It explains the remaining distribution mismatch better than requiring raw cross entropy from every training configuration to fall toward a common numerical baseline. Keep other loss terms separate before applying that interpretation.

Applying smoothing twice changes the effective amount

Suppose a data pipeline first maps a hard target to q₁=(1−ε₁)eᵧ+ε₁u, then the loss path mixes the result with the same uniform distribution u using ε₂. Expanding the expression gives q₂=(1−ε₂)(1−ε₁)eᵧ+[1−(1−ε₂)(1−ε₁)]u. The effective smoothing is therefore ε_eff=1−(1−ε₁)(1−ε₂).

Using 0.3 twice yields 0.51, rather than 0.3 or the simple sum 0.6. The three-class target changes from (0.8,0.1,0.1) to (0.66,0.17,0.17). This is algebra for mixed distributions, not an assumption that a framework necessarily applies smoothing twice. Inspect the actual data and loss paths to find whether two transformations exist.

For existing soft targets, distillation distributions or mixed examples, establish their meaning before adding another uniform mixture. Matching configuration values does not ensure matching effective targets. Soft targets are not all label smoothing, and an intentional second mixture is not automatically a design error. The calculation makes the resulting objective explicit so that it can be reviewed.

Check the objective with finite differences

The program actually executes Python standard-library arithmetic. It checks distribution conventions, boundary ε values, the reversed correct-class gradient for an overconfident prediction, finite differences of all three logits, the entropy floor, the finite optimal gap and repeated smoothing. Stable Softmax computation follows the earlier subtract-the-maximum derivation, without repeating that tutorial.

# Standard-library objective and finite differences; no PyTorch execution.
from math import exp,log,fsum,isclose,isfinite
checks=0
def check(ok):
    global checks
    assert ok
    checks+=1
def target(c,y,e):
    if c<2 or not 0<=y<c or not isfinite(e) or not 0<=e<=1:
        raise ValueError('invalid target configuration')
    return [(1-e)*(j==y)+e/c for j in range(c)]
def lp(z):
    m=max(z); a=m+log(fsum(exp(t-m) for t in z))
    return [t-a for t in z]
def ce(z,q):
    if len(z)!=len(q) or not z or not all(isfinite(t) for t in z):
        raise ValueError('invalid logits or length')
    if not all(isfinite(t) and t>=0 for t in q) or not isclose(fsum(q),1):
        raise ValueError('invalid probability target')
    return -fsum(a*b for a,b in zip(q,lp(z)))
q=target(3,0,.3)
check(all(isclose(a,b) for a,b in zip(q,[.8,.1,.1])))
check(isclose(sum(q),1))
check(target(3,0,0)==[1.,0.,0.])
check(all(isclose(t,1/3) for t in target(3,0,1)))
wrong=[.7,.15,.15]
check(not all(isclose(a,b) for a,b in zip(q,wrong)))
z=[log(.98),log(.01),log(.01)]
p=[exp(t) for t in lp(z)]
check(p[0]-q[0]>0 and p[0]-1<0)
for j in range(3):
    a=z.copy(); b=z.copy();a[j]+=1e-5;b[j]-=1e-5
    numerical=(ce(a,q)-ce(b,q))/(2e-5)
    check(isclose(numerical,p[j]-q[j],rel_tol=1e-7,abs_tol=1e-9))
opt=[log(t) for t in q]
h=-fsum(t*log(t) for t in q)
check(isclose(ce(opt,q),h) and h>0)
check(ce(z,q)>h)
check(isclose(opt[0]-opt[1],log(8)))
q2=[.7*t+.3/3 for t in q]
check(all(isclose(a,b) for a,b in zip(q2,target(3,0,.51))))
for e in (-.1,1.1,float('nan')):
    try:target(3,0,e)
    except ValueError:check(True)
    else:check(False)
for bad_z,bad_q in [([0.,0.],[1.]),([0.,0.],[.8,.8]),([0.,0.],[1.1,-.1])]:
    try:ce(bad_z,bad_q)
    except ValueError:check(True)
    else:check(False)
print(checks,'objective checks passed')
print('target',q,'minimum',h,'overconfident loss',ce(z,q))
print('double smoothing',q2,'finite logit gap',log(8))

No PyTorch call, model training or dataset evaluation was executed. The checks establish consistency between the displayed numerical objective and its derivatives. Integration should additionally compare unreduced outputs, target format and gradients from the actual loss call. Verify whether weights, ignored entries or another transformation invalidate the example’s assumptions.

The ce function rejects mismatched lengths, nonfinite logits, negative target entries and unnormalized targets. Three additional counterexamples exercise those target-format checks. It remains a single-example teaching function: batches, class weights and masks are outside its implementation. Making these conditions explicit prevents a successful finite-difference comparison from hiding an invalid target representation.

Record training and evaluation definitions separately

With q=(0.8,0.1,0.1) and prediction p=q, smoothed cross entropy is approximately 0.639032. Negative log-likelihood of the original hard label is instead −log 0.8, approximately 0.223144. The two numbers describe the same prediction but answer different questions. Do not exponentiate smoothed training loss and compare it with perplexity computed from original-label NLL as though the definitions matched.

Accuracy depends on the final class ordering and cannot alone describe these probability differences. Calibration requires its own evaluation definition. Retain ε, class count, target construction, weights, masks and reduction to distinguish a model change from a change in the measurement. The earlier pos_weight and original probabilities article studies binary weighting instead; its mechanism and correction formula are not interchangeable with smoothing.

If the target vectors, entropy floor and finite differences help locate a label-smoothing discrepancy, voluntary support through the section below is welcome. A reusable starting point is to reproduce the same objective with three classes before reconnecting it to the real training pipeline.

Sources and backfill note

CrossEntropyLoss 2.8 · Inception preprint v3, Section 7 · Preprint version history

Checked against PyTorch 2.8 definitions and the preprint revised December 11, 2015, both available before the intended slot. Original derivations and standard-library checks are not a training reproduction. Article date: October 22, 2026 at 09:00 Beijing; actual backfill time is recorded separately and does not imply earlier publication.