A language-model score rising from 88 to 90 on one set deserves investigation, but does not establish that the new system is better. Check where questions came from, how success was judged and whether observations are independent. Keep uncertainty beside the score, and preserve question-level changes when comparing systems on the same tasks.

Who is the percentage meant to represent?

Accuracy on a fixed test set describes that set. If 80 of 100 answers pass, the 80% arithmetic is unambiguous. An interval becomes relevant when using those questions to infer performance on future similar requests. That introduces assumptions about sampling, the target distribution and the evaluation protocol.

Consider a constructed support-question set containing only carefully selected short questions. Calling its score an 80% success rate for all user requests overreaches the coverage. A narrow interval cannot add missing long contexts, multi-turn state or unusual requirements. The problem is not how many decimal places are reported.

Write a checkable target first: the probability that one answer succeeds for a defined request source, time window and fixed scoring rule. Then define the sampling unit. If model, prompt, tools and retrieved data change together, a score difference cannot be attributed to just one of them. This article concerns binary success, not arbitrary continuous ratings.

The same 80% with 25 and 100 questions

Two fictional samples contain 20 successes in 25 trials and 80 in 100. Both estimate 80%. Under independent Bernoulli trials with a common success probability, nominal two-sided 95% Wilson intervals are approximately 60.9%–91.1% and 71.1%–86.7%. The second is narrower, not free of sampling uncertainty.

The 95% label describes the method’s nominal coverage in a repeated-sampling framework. It does not assign a 95% probability to the fixed true parameter being inside the particular interval already observed. Wilson is approximate, and actual coverage is not guaranteed to equal 95% at every parameter and sample size. Any population inference still needs the sampling assumptions.

More relevant observations generally reduce random uncertainty, but there is no universal rule that 100 evaluation questions make a result reliable. Task variation, the improvement worth detecting, error costs and budget influence the design. Small one- or two-percentage-point changes especially require adequate, relevant tests chosen before results are examined.

Why not just add symmetric error bars?

A familiar shortcut is p̂±1.96√[p̂(1-p̂)/n]. It is easy to calculate, but can behave poorly for small samples or proportions near zero and one. With 25 successes out of 25, it even produces zero width, which cannot justify saying the system will never fail.

The NIST statistics handbook gives the Wilson construction. Its two-sided form is convenient to implement as a center and half-width:

p_hat = k / n
denominator = 1 + z²/n
center = (p_hat + z²/(2n)) / denominator
half = z × sqrt(p_hat(1-p_hat)/n + z²/(4n²)) / denominator
interval = [center - half, center + half]
z ≈ 1.959964   # nominal two-sided 95%

Here k is the number of successes, n the number of trials, and z corresponds to the chosen nominal two-sided level. The center need not equal the observed rate, so the interval need not be symmetric around 80%. With 25 successes, the Wilson lower bound is about 86.7%; with 25 failures, the upper bound is about 13.3%.

These are method examples, not model measurements. Exact binomial intervals and other approaches may suit particular small-sample or rare-failure settings. State the choice with the inference target rather than selecting whichever method looks most favorable after seeing results.

Output counts alongside intervals

import math

def wilson95(k, n):
    if type(k) is not int or type(n) is not int:
        raise ValueError("Counts must be integers")
    if n <= 0 or not 0 <= k <= n:
        raise ValueError("Expected 0 <= k <= n and n > 0")
    z = 1.959963984540054
    p = k / n
    denominator = 1 + z * z / n
    center = (p + z * z / (2 * n)) / denominator
    half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / denominator
    low = 0.0 if k == 0 else max(0.0, center - half)
    high = 1.0 if k == n else min(1.0, center + half)
    return low, high

def paired_counts(rows):
    gains = losses = same = 0
    for a, b in rows:
        if a not in (0, 1) or b not in (0, 1):
            raise ValueError("Expected binary scores")
        if a == b:
            same += 1
        elif b == 1:
            gains += 1
        else:
            losses += 1
    return gains, losses, same

for k, n in ((20, 25), (80, 100), (0, 25), (25, 25)):
    low, high = wilson95(k, n)
    assert 0 <= low <= k / n <= high <= 1
    print(k, n, f"[{low:.4f}, {high:.4f}]")
small = wilson95(20, 25)
large = wilson95(80, 100)
assert large[1] - large[0] < small[1] - small[0]
assert wilson95(0, 25)[0] == 0
assert wilson95(25, 25)[1] == 1
for bad in ((0, 0), (26, 25), (-1, 25), (1.5, 25)):
    try:
        wilson95(*bad)
    except ValueError:
        pass
    else:
        raise AssertionError("Invalid counts accepted")

rows = [(1, 1)] * 84 + [(0, 0)] * 6 + [(1, 0)] * 4 + [(0, 1)] * 6
assert len(rows) == 100
assert sum(a for a, b in rows) == 88
assert sum(b for a, b in rows) == 90
assert paired_counts(rows) == (6, 4, 90)
print("gains, losses, unchanged:", paired_counts(rows))

try:
    paired_counts([(1, 2)])
except ValueError:
    pass
else:
    raise AssertionError("Nonbinary score accepted")

This original Python code was executed, checking both 80% examples, their different widths, all-success and all-failure cases, and invalid counts. Its intervals are approximately [0.6087,0.9114], [0.7112,0.8666], [0,0.1332] and [0.8668,1]. A Wilson calculation uses a constant number of arithmetic operations. The subsequent paired tally takes O(n) time and O(1) extra space for n records, excluding stored inputs.

Reports should retain k, n, method, nominal level and scoring protocol rather than only a percentage with an error bar. Handling numerical boundaries does not validate independence, judging quality or representativeness. The code cannot identify those design failures automatically.

Return to the 100 questions behind 88 and 90

Use a separate teaching example. Systems A and B answer the same 100 questions: both pass 84, both fail 6, A alone passes 4 and B alone passes 6. A therefore scores 88 and B scores 90. The net gain of two percentage points contains six gains and four regressions, not merely two changed answers.

This makes the next investigation concrete. Inspect whether gains match the intended requirements and whether regressions concentrate in critical categories. If all four losses concern behavior that must be retained, the aggregate score hides a deployment concern. Release decisions still need predefined constraints rather than letting the average decide.

Statistical comparisons should also preserve pairing. Treating A and B outcomes on the same question as unrelated discards information. Overlap or nonoverlap between separate intervals does not replace a test or difference interval suited to the design. The diagram and code provide a descriptive breakdown only, with no significance conclusion.

Repeating one question ten times does not create ten questions

Repeated generation can reveal a question’s stability under sampling randomness, but does not expand task coverage. Questions from one conversation or document can share information or difficulty too. Treating all correlated records as independent makes a large n look more informative than it is.

Separate two goals: estimating repeated success on one question, and estimating performance across different future requests. The first concerns that question and generation settings; the second also requires coverage of request variation. When conversations or documents form groups, use an analysis matching that grouping rather than burying dependence in a total.

Automated grading adds another source of error. If the judge favors length, B can earn more points merely by becoming verbose. Freeze the rubric and manually review representative passes, failures and disagreements before polishing a fourth decimal place. For diagnosing failure stages, combine this with the RAG evaluation breakdown.

Do not let failures silently change the denominator

An evaluation record should connect a question ID, input version, model configuration, output, score and execution status. Define the success-rate denominator before running. Preserve timeouts, API errors, parsing failures and refusals as explicit states. Exporting only answers that parsed successfully removes some of the most important requests from the report.

Whether infrastructure failures count as task failures depends on the target. An end-to-end service evaluation can include timely delivery in its success definition. A study isolating model behavior can instead specify infrastructure exclusions in advance. Report total requests, scored requests and exclusions for either protocol. Do not switch definitions after an inconvenient result appears.

Repeated attempts also need a fixed budget. Success on one answer and success on any of several attempts are different metrics. Increasing the number of chances can raise the latter while also increasing calls and latency. Comparing several chances for the new system against one chance for the old system does not isolate a model improvement. See the definition of pass@k for the related distinction.

Preserving failures has an engineering payoff beyond interval calculation. It helps identify whether the next change belongs in the model, prompt, parser or delivery chain. Without that evidence, a larger sample may only describe an ambiguous metric more precisely.

Leave an evaluation someone can audit

For a prompt or retrieval change, retain a frozen test set, question IDs, A/B versions and settings, outputs and scoring reasons. Separate development data from final evaluation. Once repeated inspection and tuning use a set to select a system, it is no longer an untouched check of that selection.

Choose the primary metric and critical categories in advance, inspect regressions, then assess gains. Accuracy sits alongside latency, cost, format requirements and coverage. The original HELM paper emphasizes standardized conditions and multiple metrics. It first appeared in 2022 and was published in TMLR in 2023. This article uses that evaluation perspective rather than presenting rankings or calling it a new paper.

If one habit survives, make every impressive percentage traceable to a denominator and underlying evidence. That helps distinguish a change on these questions, a plausible benefit for target requests, and evidence adequate for release. An interpretable score is a better foundation for deciding where to spend the next round of engineering effort.

阅读中文版

Sources

• NIST — Confidence intervals for proportions
• Holistic Evaluation of Language Models — 2023 manuscript

Publication note: backfilled on the evening of October 14, 2026, with the planned 09:00 Beijing article date retained. Sources predate that slot; diagram values are teaching examples.