A lower Brier score does not, by itself, prove better probability calibration. The squared-error score reflects both probability mismatch and the ability to distinguish cases. A model that separates positives from negatives can score better even when its probability groups show larger discrepancies. To examine whether events assigned 80% probability occur about 80% of the time, inspect the groups and retain their sample counts.

Fix the score convention first

This article uses binary labels y in {0,1} and the probability p of the positive class. Define BS as the mean of (p−y)²; lower is better. We calculate one squared error for the positive-class probability, rather than summing errors for both classes. The latter binary convention gives twice the result. Establish this convention and any sample weights before comparing reports.

Accuracy looks only at which side of a threshold a prediction falls on. Brier also measures its distance from the observed label. Reporting 0.9 for a positive case incurs less squared error than reporting 0.6, but one positive outcome does not justify always predicting 0.9. Pushing every prediction toward zero or one incurs larger penalties when it is wrong. Probability evaluation requires a comparable collection of forecasts.

The scikit-learn 1.5 calibration guide distinguishes reliability, resolution and uncertainty, and warns against isolating calibration quality from the Brier total alone. The following original discrete example derives the issue. It reproduces neither the documentation’s experiments nor a trained model, and presents an established method rather than new research.

Group cases with the same predicted probability

Within a group, every case has prediction p and the observed positive fraction is q. Write p−y as (p−q)+(q−y), square it and average within that group. The cross term vanishes because the mean of q−y is zero. The result is (p−q)²+q(1−q). The first term measures mismatch between the prediction and the group’s observed fraction; the second describes variation of the labels within that group.

Let the overall positive fraction be q̄ and weight each group by its fraction of all samples. The exact identity is BS=REL−RES+UNC. Here REL is the sum of group weight times (p−q)², RES is the sum of group weight times (q−q̄)², and UNC=q̄(1−q̄). The group weights sum to one; there is no additional division by the number of groups.

Smaller REL means closer agreement within these groups. Larger RES means that group outcome fractions depart more from the overall fraction, distinguishing cases with different observed risks. It is subtracted from the total. UNC depends on label prevalence in this evaluation set. When two models use the same set, UNC stays fixed, but REL and RES can both change.

A lower total with larger group discrepancies

Construct 20 cases: ten negatives and ten positives. Scheme A predicts 0.5 for all of them. Its single prediction group has an observed positive fraction of 0.5, so its empirical REL is zero. It distinguishes no cases: RES is zero and BS is 0.25.

Scheme B assigns 0.1 to the ten negatives and 0.9 to the ten positives. Every prediction differs from its label by 0.1, giving BS=0.01. However, the observed positive fractions are zero in the 0.1 group and one in the 0.9 group. Both have a discrepancy of 0.1, producing REL=0.01, larger than scheme A’s. The lower total comes from RES rising to 0.25, not from a reduction in REL.

These predictions were assigned deliberately to construct a counterexample. They are not outputs from training and establish no model’s generalization performance. Scheme A’s agreement on this finite sample does not prove population calibration on new data. The example refutes the implication that lower total loss necessarily means better calibration; it does not establish constant prevalence prediction as the best forecasting strategy.

A coarse bin can cancel opposing discrepancies

Merge all of scheme B into one diagnostic bin. Its mean prediction is 0.5 and its positive fraction is also 0.5. The plotted point lies on the diagonal, hiding discrepancies in opposite directions. Retaining the 0.1 and 0.9 groups reveals their observed fractions of zero and one.

Report the binning rule and counts alongside a reliability diagram. Very fine bins can have noisy observed fractions; coarse bins can conceal local differences. Repeatedly changing the number of bins until the picture looks favorable is not independent evidence. Fix the comparison rule in advance and examine sensitivity to other reasonable choices.

The exact decomposition above groups identical prediction values. With continuous scores, interval bins contain different p values. Replacing them by their bin mean does not justify using the same residual-free identity without qualification. Calculate the original BS per sample and treat the binned plot as another diagnostic. Discrete predictions make every step of this example directly checkable.

Run the exact-arithmetic check

Save this as brier_example.py and run python3 brier_example.py without -O, which disables assertions. It was executed with Python 3.15.0. The standard-library Fraction type avoids display rounding in the identity check. Inputs are aligned binary labels and exact probabilities; the result order is BS, REL, RES, UNC.

from collections import defaultdict
from fractions import Fraction as F


def evaluate(labels, probabilities):
    if not labels or len(labels) != len(probabilities):
        raise ValueError('nonempty aligned inputs required')
    if any(y not in (0, 1) for y in labels):
        raise ValueError('binary labels required')
    if any(not 0 <= p <= 1 for p in probabilities):
        raise ValueError('probabilities must be in [0, 1]')
    n = len(labels)
    prevalence = F(sum(labels), n)
    groups = defaultdict(list)
    loss = F(0)
    for y, p in zip(labels, probabilities):
        groups[p].append(y)
        loss += (p - y) ** 2
    reliability = resolution = F(0)
    for p, ys in groups.items():
        weight = F(len(ys), n)
        rate = F(sum(ys), len(ys))
        reliability += weight * (p - rate) ** 2
        resolution += weight * (rate - prevalence) ** 2
    uncertainty = prevalence * (1 - prevalence)
    brier = loss / n
    assert brier == reliability - resolution + uncertainty
    return brier, reliability, resolution, uncertainty


# Synthetic assigned predictions, not a trained classifier or benchmark.
y = [0] * 10 + [1] * 10
constant = [F(1, 2)] * 20
separating = [F(1, 10)] * 10 + [F(9, 10)] * 10
less_extreme = [F(1, 5)] * 10 + [F(4, 5)] * 10
assert evaluate(y, constant) == (F(1, 4), F(0), F(0), F(1, 4))
assert evaluate(y, separating) == (F(1, 100), F(1, 100), F(1, 4), F(1, 4))
assert evaluate(y, less_extreme)[0] == F(1, 25)
assert all((p >= F(1, 2)) == bool(label)
           for label, p in zip(y, separating))
assert all((p >= F(1, 2)) == bool(label)
           for label, p in zip(y, less_extreme))
# Merging everything into one diagnostic bin hides opposite errors.
mean_p = sum(separating, F(0)) / len(separating)
assert mean_p == F(sum(y), len(y)) == F(1, 2)
for labels, probs in [([], []), ([0, 1], [F(1, 2)]), ([0], [F(2)])]:
    try:
        evaluate(labels, probs)
    except ValueError:
        pass
    else:
        raise AssertionError('invalid input accepted')
for name, probs in [('constant', constant), ('separating', separating)]:
    print(name, tuple(str(x) for x in evaluate(y, probs)))
print('one-bin mean prediction and positive rate:', mean_p)

The two outputs are (1/4,0,0,1/4) and (1/100,1/100,1/4,1/4). A further scheme predicts 0.2 and 0.8. It and the 0.1/0.9 scheme classify every case correctly at threshold 0.5, yet their BS values are 0.04 and 0.01. These are different assigned probabilities on the same cases, not before-and-after results from calibration training. Classification accuracy cannot replace probability scoring. The program also rejects empty inputs, mismatched lengths and out-of-range probabilities, preventing silent sample loss or invalid values from changing the calculation.

Protect the comparison before fitting a calibrator

Retain per-case predictions, labels, grouping rules and counts. Report BS and a reliability diagram on the same target set. If prevalence changes, UNC changes too; different dates or populations should not be treated automatically as model-version effects. Keep a consistent rule for excluding cases with unavailable labels.

If fitting a temperature, sigmoid or another calibration mapping, distinguish model-training data, calibration-fitting data and final-evaluation data. Tuning a mapping against final-evaluation labels and claiming improvement on those same labels confuses fitting with generalization. Small groups provide uncertain diagnostics, not a stable promise merely because their points lie near a diagonal.

The site’s PBC weather forecast explanation focuses on valid distributions, scoring baselines and data availability. This article adds the binary BS decomposition and the coarse-bin cancellation counterexample. 中文版. If this derivation and runnable example help you inspect a probability-evaluation report, voluntary support is welcome.

The original article date is October 28, 2026 at 23:30 Beijing time; actual backfill is October 30, 2026. The cited version 1.5 documentation predates that slot: version 1.5.2 was released in September 2024. No real model training, calibrator fitting or benchmark improvement is claimed.