In single-output regression, predictions shaped [N,1] and targets shaped [N] should not be subtracted merely because both contain N numbers. Broadcasting can create an [N,N] difference matrix: every prediction is compared with every target instead of its own target.

This changes the objective, not just the amount of work. The wrong objective pulls outputs toward the batch target mean. A perfectly correct prediction can then have loss two, while a constant prediction has a smaller loss. The derivation and pairing tests below make that failure reproducible. 中文版.

Align dimensions from the right

The broadcasting rules align trailing dimensions, allowing equal sizes or a size of one. Treat [N] as [1,N]. Subtracting it from [N,1] produces [N,N], with row i selecting a prediction and column j selecting a target.

The MSELoss contract calls for targets with the same shape as the input. This article derives the effect through explicit squared differences rather than depending on whether a particular release warns. A valid mathematical broadcast does not establish the pairing a task requires.

Both [N] and [N,1] can be valid conventions when used consistently. Multi-output regression [N,D] has a different contract. Reshaping two arrays into equal element counts merely to avoid an error cannot validate their axis meanings.

A correct prediction gets a nonzero wrong loss

Take predictions p=[1,3] and targets y=[1,3]. Paired squared errors are [0,0], giving mean zero. All-pairs broadcasting produces [[0,4],[4,0]], whose four-element mean is two.

Swap only the labels to [3,1]. Correctly paired MSE becomes four because each prediction now has the wrong target. The all-pairs mean remains two: it retains the same collection of target values but loses which one belongs to each input.

A finite scalar loss, working backward pass or declining curve therefore cannot establish correct supervision. A wrong objective can have a well-defined minimum that an optimizer steadily approaches. What must be tested is whether the intended input-to-target correspondence survives.

Why the batch mean replaces each target

The paired objective is L_pair=(1/N)Σᵢ(pᵢ−yᵢ)², with derivative 2(pᵢ−yᵢ)/N for pᵢ. Each output follows its own target. The all-pairs objective is L_all=(1/N²)ΣᵢΣⱼ(pᵢ−yⱼ)², whose derivative is 2(pᵢ−ȳ)/N, using batch target mean ȳ.

If each prediction in this fixed batch is a freely chosen variable, every optimum is therefore ȳ. Here that mean is two. Predictions [2,2] have all-pairs loss one, smaller than the erroneous loss two for the correct [1,3]. Optimizing the wrong objective can actively damage an initially correct prediction.

Equivalently, for fixed pᵢ, the average over targets satisfies (1/N)Σⱼ(pᵢ−yⱼ)²=(pᵢ−ȳ)²+Var_batch(y), where variance divides by N. The variance term does not depend on the prediction. The remaining term pulls it toward the mean without recovering label correspondence.

A real network shares parameters across outputs, and its batch distribution and optimization affect the eventual fit. The free-variable optimum is not a guaranteed training outcome. It identifies the supervision change precisely: yᵢ has been replaced by a batch mean, and additional capacity does not reveal the programmer’s original intention.

Nine exact checks expose pairing semantics

The code implements paired and all-pairs objectives using Fraction. It does not import PyTorch or measure device performance. Each function rejects empty inputs outside its supported contract so an undefined empty mean does not distract from the pairing question.

# Exact paired and all-pairs objectives, not a PyTorch execution.
from fractions import Fraction as F

def paired(pred, target):
    if not pred or len(pred)!=len(target):
        raise ValueError("equal nonempty lengths required")
    return sum((F(p)-t)**2 for p,t in zip(pred,target))/len(pred)

def all_pairs(pred, target):
    if not pred or not target:
        raise ValueError("nonempty inputs required")
    return sum((F(p)-t)**2 for p in pred for t in target)/(len(pred)*len(target))

p=[F(1),F(3)]; y=[F(1),F(3)]
assert paired(p,y)==0
assert all_pairs(p,y)==2
assert paired(p,list(reversed(y)))==4
assert all_pairs(p,list(reversed(y)))==2
assert all_pairs([F(2),F(2)],y)==1
assert paired([F(2),F(2)],y)==1
assert all_pairs([F(1)],[F(1)])==paired([F(1)],[F(1)])
assert len([(a,b) for a in p for b in y])==4
assert paired(p,y)!=all_pairs(p,y)
print("9 MSE pairing checks passed")

All nine assertions passed locally. They cover correct predictions, label permutation, constant predictions, the single-sample degeneracy and pair counts. N=2 is intentional: at N=1 the objectives coincide, so a one-sample-only test can hide this shape error. Allowing different nonempty lengths in the mathematical all-pairs function does not mean a production regression API should accept mismatched sample counts.

Choose squeeze behavior after choosing semantics

For a verified single-output prediction [N,1], a [N] target convention can remove that specific final singleton dimension. Alternatively, a [N,1] target convention can add the matching target dimension. First confirm that the axis represents one regression output, rather than a sequence or channel dimension that happens to equal one.

An unrestricted squeeze removes every singleton axis. With batch size one it may remove the batch axis too, creating different ranks in training and evaluation paths. Assert exact prediction and target shapes at the loss boundary instead of relying on an operation that seems to remove all unwanted dimensions.

Comparing numel is insufficient. [N,D] and [D,N] can contain the same number of elements while exchanging sample and output meanings. Reshape does not validate sample order. Transposition, sorting and concatenation also require explicit correspondence checks in the data pipeline.

Use label permutation as a deliberate negative test

Create three distinct targets and predictions equal to them element by element. Paired loss should be zero. Permute only the targets while holding inputs and predictions fixed; the loss should change according to the selected values. If it does not, inspect accidental all-pairs calculation or an earlier loss of alignment.

By contrast, jointly permuting inputs, predictions and labels should preserve the paired mean. One test breaks correspondence and the other preserves it. Their purposes differ, and keeping both avoids a repair that merely matches one attractive scalar result.

Also cover batch size one, larger batches, a short final batch, multiple outputs and the chosen empty-data policy. Dropping every short final batch can hide a shape problem by changing data selection. Keep synthetic sample identifiers beside their targets in the test; public teaching examples do not need real user records.

If all labels in a test are identical, breaking correspondence may also leave its loss unchanged. Choose distinct synthetic labels deliberately; permutation invariance alone is not a universal bug detector. The expected paired values should be calculated explicitly.

Place shape assertions at the loss boundary

Check rank, axis sizes and label meaning near the loss call, where an error is easiest to explain. Broadcasting everything into one final scalar discards much of the evidence needed for diagnosis.

This is distinct from gradient-accumulation reduction, which asks how valid losses are combined across microbatches. First establish which samples one loss compares, then examine averaging and accumulation. Otherwise aggregation can be perfectly accurate for the wrong objective.

If the derivation and label-permutation tests help eliminate a cause of declining loss with poor regression results, voluntary support through the section below is welcome. The reusable outcome is an executable constraint on sample correspondence.

Sources and backfill note

Broadcasting semantics、MSELoss

The definitions were checked against PyTorch 2.8 material available before the planned slot. Derivations, counterexamples and figures are original teaching material. The article keeps October 19, 2026 at 19:00 Beijing time as its planned date and was backfilled that evening. Standard-library code ran locally; no PyTorch model or device-performance benchmark ran.