When A and B answer the same evaluation cases, a paired bootstrap should apply the same sampled case indices to both. Resampling A and B independently breaks the same-case relationship and changes the variability of their difference. It can produce nonzero differences even when their scores match on every case.

We will use four synthetic cases and exact rational arithmetic to inspect both designs. The subject is the resampling unit and index relationship, not real model rankings, a significance claim, or a deployment recommendation. 中文版.

Align cases before discussing resampling

Let aᵢ and bᵢ be binary scores on the same case i, with one meaning success and zero failure. For n cases, the observed accuracy difference is Δ = Σ(aᵢ − bᵢ)/n. This descriptive value needs no resampling. Bootstrap examines how the statistic changes under a specified resampling design.

In one paired replicate, draw n indices J with replacement from zero through n−1, then compute both A[J] and B[J]. A case may occur repeatedly or be absent, but each occurrence carries both scores. The SciPy 1.16.0 bootstrap documentation explicitly uses shared indices for paired=True and independent resampling for paired=False. The program below checks the idea without calling SciPy.

Sharing indices is more explicit than hoping that two generators with the same seed remain aligned. Generate the index sequence as data and apply it to all paired fields. Extra random calls, unequal lengths, or different filtering can make apparently identical seed settings select different cases.

Identical case scores can acquire a difference

Set A = B = (1, 1, 0, 0). Both accuracies are one half, and the scores agree case by case, not merely in total. Every shared index sequence gives equal resampled means, so the paired difference is always zero.

With independent draws, A could receive indices (0, 0, 0, 0), giving mean one, while B receives (2, 2, 2, 2), giving mean zero. The difference is one. This is not an observed model divergence: the resampling assigned different case mixtures. Independent designs can be appropriate elsewhere, but they do not preserve the same-case comparison intended here.

Drawing four indices from four cases gives 4⁴ = 256 equally likely ordered sequences. The paired design computes one difference per sequence. The independent design has 256² = 65,536 sequence pairs. The code counts resampled means and multiplies their frequencies to count those independent combinations exactly, without a seed or simulation convergence requirement.

Covariance explains both directions of distortion

For the fixed rows, define Var_emp(A) = Σ(aᵢ − ā)²/n and Cov_emp(A,B) = Σ(aᵢ − ā)(bᵢ − b̄)/n. These empirical-distribution quantities use denominator n, rather than the n−1 denominator of an unbiased sample variance.

A replicate draws n independent indices while sharing each draw across A and B. The conditional resampling variance of the mean difference is [Var_emp(A) + Var_emp(B) − 2Cov_emp(A,B)]/n. Independent resampling removes the covariance term, leaving [Var_emp(A) + Var_emp(B)]/n. This is a derivation about the empirical resampling distribution, not an exact estimate of a population difference.

For identical rows, both variances and the covariance are 1/4. The paired variance is zero; the independent variance is 1/8. Now set B = (0, 0, 1, 1), complementary to A case by case. Covariance becomes −1/4, paired variance becomes 1/4, and independent variance remains 1/8. Breaking pairing can therefore enlarge or shrink variability. It is not uniformly conservative.

In both constructions, A and B each have two successes and marginal accuracy one half. Only their same-case relationship changes, yet the paired difference variance moves from zero to 1/4. Two aggregate scores cannot reconstruct this distinction. Evaluation exports therefore need stable case identifiers and both per-case score columns, not just averages.

Twenty exact checks

The program uses the Python standard library and was run locally on Python 3.15.0. Fraction keeps means and variances exact, without Monte Carlo error. The functions accept only binary scores and at most six rows, preventing this exponential teaching enumerator from being mistaken for a practical large-evaluation implementation.

from collections import Counter
from fractions import Fraction as F
from itertools import product


def validate(a, b):
    if not a or len(a) != len(b):
        raise ValueError('nonempty, equally sized aligned inputs required')
    if len(a) > 6:
        raise ValueError('exact teaching enumerator is limited to six rows')
    if any(x not in (0, 1) for x in (*a, *b)):
        raise ValueError('this example accepts binary scores only')


def sample_means(values):
    n = len(values)
    return Counter(F(sum(values[i] for i in idx), n)
                   for idx in product(range(n), repeat=n))


def paired_dist(a, b):
    validate(a, b)
    n = len(a)
    return Counter(F(sum(a[i] - b[i] for i in idx), n)
                   for idx in product(range(n), repeat=n))


def independent_dist(a, b):
    validate(a, b)
    out = Counter()
    means_a, means_b = sample_means(a), sample_means(b)
    for ma, ca in means_a.items():
        for mb, cb in means_b.items():
            out[ma - mb] += ca * cb
    return out


def moments(dist):
    total = sum(dist.values())
    mean = sum((x * count for x, count in dist.items()), F(0)) / total
    var = sum(((x - mean) ** 2 * count for x, count in dist.items()), F(0)) / total
    return mean, var


checks = 0

def check(condition):
    global checks
    assert condition
    checks += 1


a = (1, 1, 0, 0)
p = paired_dist(a, a)
u = independent_dist(a, a)
check(p == Counter({F(0): 256}))
check(sum(u.values()) == 65536)
check(moments(p) == (0, 0))
check(moments(u) == (0, F(1, 8)))
check(u[F(1)] == 256 and u[F(-1)] == 256)
check(u[F(0)] == 17920)
check(u == Counter({-x: count for x, count in u.items()}))

b = (0, 1, 0, 0)
d = paired_dist(a, b)
expected = {F(0): 81, F(1, 4): 108, F(1, 2): 54,
            F(3, 4): 12, F(1): 1}
check(d == Counter(expected))
check(sum(d.values()) == 256)
check(moments(d) == (F(1, 4), F(3, 64)))
check(F(d[F(0)], 256) == F(81, 256))
check(F(sum(c for x, c in d.items() if x > 0), 256) == F(175, 256))
check(paired_dist(b, a) == Counter({-x: c for x, c in d.items()}))
check(paired_dist(tuple(a[i] for i in (2, 0, 3, 1)),
                  tuple(b[i] for i in (2, 0, 3, 1))) == d)
anti = tuple(1 - x for x in a)
check(moments(paired_dist(a, anti)) == (0, F(1, 4)))
check(moments(independent_dist(a, anti)) == (0, F(1, 8)))
for bad_a, bad_b in [((), ()), ((1,), (1, 0)), ((2,), (0,)),
                     ((1,) * 7, (0,) * 7)]:
    try:
        paired_dist(bad_a, bad_b)
    except ValueError:
        check(True)
    else:
        raise AssertionError('expected ValueError')
print(f'{checks} exact-resampling checks passed')
print('identical paired variance:', moments(p)[1])
print('identical independent variance:', moments(u)[1])
print('one-row improvement counts:', sorted(d.items()))
print('positive fraction:', F(175, 256))

The local output was “20 exact-resampling checks passed”. Assertions cover identical scores, complementary scores, and a one-case improvement. They check sign reversal when A and B are swapped and distribution invariance when both columns are reordered together. Empty inputs, different lengths, nonbinary scores, and inputs beyond the example’s size limit are rejected.

Exact means exact counting within this specified empirical resampling space. It is not a guarantee about actual user requests, future model scores, or confidence-interval coverage. Enumeration is useful as a small reference calculation and regression check; nⁿ enumeration is not a suitable implementation for a normal evaluation set.

Zero variance for identical rows also belongs only to the observed empirical resampling distribution. It does not prove that two real models agree on every future input. If unseen cases separate them, resampling the old rows cannot discover those cases. Exact enumeration removes calculation error, not missing evaluation coverage.

A positive fraction is not a probability of being better

Keep A = (1, 1, 0, 0) and use B = (0, 1, 0, 0). The observed difference is 1/4, with case differences (1, 0, 0, 0). In four draws, case zero can occur zero through four times. Resampled differences are therefore limited to zero, 1/4, 1/2, 3/4, and one.

Their respective counts are 81, 108, 54, 12, and one, summing to 256. Missing case zero entirely has probability (3/4)⁴ = 81/256. The positive-difference fraction is 175/256, approximately 68.36%. The program reproduces these numbers directly.

Do not relabel 68.36% as the probability that A is better in the real world or directly as a null-hypothesis p-value. No null distribution was constructed, no confidence-interval method was selected, and no adjustment for repeatedly selecting a favorable configuration was made. Four cases expose an indexing mechanism; they are not a recommended model-evaluation sample size.

Case identity and the sampling unit matter

Equal column lengths do not establish correct alignment. Match A and B by stable case identifiers first, then apply a defined policy to missing outputs, timeouts, and scoring failures. Independently deleting failed rows and pairing by array position can attach scores from different cases. Shared indices preserve positions; they cannot repair incorrect identity matching.

Also examine whether cases are suitable independent sampling units. Several cases derived from one document or conversation can share context and be dependent. A simple case-level bootstrap does not automatically capture that structure. A grouped design needs an explicit definition of how documents or conversations are sampled and weighted before choosing the implementation.

Multiple generations per model and case do not simply create more independent cases. Coverage across cases and variability from generation are different sources of uncertainty. This example fixes one binary score per case and does not estimate generation variability. Studying that question requires retaining the repeated-run structure rather than merely changing a paired option.

Separate the statistic from the inference report

For a mean metric, resampling per-case differences directly computes the same mean difference. For a nonlinear statistic such as overall F1, recompute the target statistic on the selected data rather than assuming an average of per-case F1 values equals overall F1. Both the resampling design and statistic must match the evaluation definition.

The earlier article on LLM evaluation and Wilson intervals discusses an individual success-rate interval and case-level change records. This article adds the resampling mechanism for a difference between two systems: preserve the correct pairing before choosing an uncertainty report. Paired bootstrap cannot repair a biased case set, incorrect scoring, or evaluation leakage.

The shared-index interface was checked against the SciPy 1.16.0 documentation. Both diagrams, the three four-case constructions, variance derivations, and twenty checks are original teaching material. No model or SciPy run was performed, and no confidence interval is reported. The article retains October 23, 2026 at 19:00 Beijing time and was actually published as a catch-up article on October 26, 2026.