itertools.groupby: Why a Query Appears in Two Groups

黎 浩然/ 24 10 月, 2026/ 计算机/COMPUTER/ 0 comments

itertools.groupby does not automatically merge every occurrence of a key across its input. It groups adjacent records with equal keys. Q1, Q2, Q1 still produces three groups. If a dictionary then stores one summary per group, the second Q1 can overwrite the first, allowing evaluation to finish normally while losing an attempt.

Contents
  1. Do you need consecutive runs or global groups?
  2. Three records turn 3/4 into 1/2
  3. A subgroup is not a snapshot for later reading
  4. Choose a repair from the input contract
  5. Run the counterexample and iterator checks
  6. Put ordering and counts in the evaluation interface

Do you need consecutive runs or global groups?

The official CPython v3.10.0 documentation source describes consecutive grouping and the shared input iterator used by the outer object and its subgroups. This fixed version predates the original article slot. The local example runs on Python 3.15.0; the behavior is an established interface, not a new feature.

If a log is already clustered by query, with each query occupying a single run, groupby is useful for consuming one run at a time. If records arrive in asynchronous completion order, attempts for the same query may be separated by other queries. Completion order and clustering by query are different input contracts. The function neither searches backward nor knows which future records belong to the current query. A changed key starts another group.

Sorting is one way to make equal keys adjacent, but it is not necessary for groupby to operate. Q2, Q2, Q1, Q1 also has one run per query despite not being in ascending order. Treating “sort first” as the entire rule can overlook an already clustered input and hide the cost of sorting before a supposedly streaming operation.

Three records turn 3/4 into 1/2

Construct three fictional evaluation records: (Q1,1), (Q2,1), (Q1,0). The second field marks an attempt as successful or unsuccessful. These are assigned outcomes, not measurements of a real model. Our target is to average attempts within each query, then give the two queries equal weight: Q1 scores 1/2, Q2 scores 1, and the final mean is 3/4.

Direct grouping produces three runs: one successful Q1 attempt, one successful Q2 attempt and one failed Q1 attempt. A dictionary comprehension storing (total,count) by query replaces Q1’s (1,1) with (0,1). Two query keys remain, so the key count looks plausible. However, one Q1 success has disappeared and the final mean becomes 1/2.

Completion-order input
Q1 success → Q2 success
→ Q1 failure
Consecutive runs
Q1:[1] │ Q2:[1] │ Q1:[0]
Dictionary assignment
Q1:(1,1) replaced by (0,1)
Wrong query mean: 1/2
Global aggregation
Q1:(1,2), Q2:(1,1)
Correct query mean: 3/4
Figure 1. Original three-record counterexample. groupby preserves the three runs; later assignment to an existing key overwrites information. These are separate steps.

Checking that the final dictionary contains two queries is insufficient. Also reconcile the attempt counts: their correct sum is three, while the defective summary counts two. The success total drops from two to one. An additional assertion verifies that the defective mean is exactly 1/2, making the reported fraction directly reproducible. Key count, record count and success count are different checks; one passing does not establish the other two.

A subgroup is not a snapshot for later reading

Subgroup lifetime is a separate issue. Take the stream A, A, B. Obtain the first group and consume one A, then advance the outer iterator to the B group. The remaining A has been consumed while finding the next group. Reading the old subgroup now yields nothing. Saving its iterator object did not save its contents.

Time 1: obtain A group
Shared input A A B
Consume only the first A
Time 2: advance outer iterator
Pass the remaining A
Discover the B group
Time 3: read old A group
Result is empty
Materialize while it is current
Figure 2. Original shared-cursor timeline. Outer iteration and subgroups advance through the same input; an old subgroup is not an independently replayable collection.

If later work needs all records, call list(group) while that group is current and store the list. If it needs only a total and count, aggregate immediately instead of retaining every row just to calculate a mean. Both choices address consumption timing. Neither automatically combines equal-key runs that occur at different input positions.

Choose a repair from the input contract

For finite input, sort with the same key function before grouping. This makes the Q1 records adjacent and suits a batch that already exists in memory, especially when later operations need each group’s records. However, sorted consumes the input and stores its result. Sorting followed by iteration is not a constant-memory strategy for arbitrarily large data.

For mergeable summaries such as totals and counts, accumulating by key in a dictionary is more direct and does not depend on arrival order. With N records and U distinct keys, hash accumulation typically takes average O(N) time and O(U) summary state. Sorting typically takes O(N log N) time and stores O(N) records. The dictionary retains only total and count; storing all rows instead would no longer require just O(U) space. These bounds abstract away integer size and key-comparison costs; they are not measured service latency.

If upstream promises a single run for each key, a seen set can reject a key that reappears after its run ended. This validates clustering, not ascending order, and requires O(U) remembered keys. It also does not roll back previously consumed input or externally written results. A final exception cannot establish that an external report received no partial updates.

Run the counterexample and iterator checks

Save this as groupby_example.py and run python3 groupby_example.py without -O, which disables assertions. It was executed with Python 3.15.0. The checks cover interleaved runs, overwritten summaries, correct sorted and hash aggregation, counts, empty input, rejection of a repeated run, and exhaustion of an old subgroup after outer iteration advances.

from collections import defaultdict
from fractions import Fraction as F
from itertools import groupby
from operator import itemgetter


def stats(group):
    total = count = 0
    for _, success in group:
        total += success
        count += 1
    return total, count


def hash_totals(rows):
    result = defaultdict(lambda: [0, 0])
    for query, success in rows:
        result[query][0] += success
        result[query][1] += 1
    return {key: tuple(value) for key, value in result.items()}


def checked_runs(rows):
    seen = set()
    for query, group in groupby(rows, key=itemgetter(0)):
        if query in seen:
            raise ValueError('query reappeared after its run ended')
        seen.add(query)
        yield query, stats(group)


# Assigned outcomes of fictional query attempts, not model measurements.
rows = [('Q1', 1), ('Q2', 1), ('Q1', 0)]
runs = [(key, list(group)) for key, group in groupby(rows, key=itemgetter(0))]
assert [key for key, _ in runs] == ['Q1', 'Q2', 'Q1']
bad = {key: stats(group) for key, group in groupby(rows, key=itemgetter(0))}
assert bad == {'Q1': (0, 1), 'Q2': (1, 1)}
bad_macro = sum((F(t, n) for t, n in bad.values()), F(0)) / len(bad)
assert bad_macro == F(1, 2)
ordered = sorted(rows, key=itemgetter(0))
fixed = dict(checked_runs(ordered))
assert fixed == hash_totals(rows) == {'Q1': (1, 2), 'Q2': (1, 1)}
macro = sum((F(total, count) for total, count in fixed.values()), F(0)) / len(fixed)
assert macro == F(3, 4)
assert sum(x[0] for x in fixed.values()) == 2
assert sum(x[1] for x in fixed.values()) == len(rows)
assert F(2, 3) != macro  # Attempt-weighted and query-weighted means differ.
assert hash_totals([]) == {} and list(checked_runs([])) == []
try:
    list(checked_runs(rows))
except ValueError:
    pass
else:
    raise AssertionError('noncontiguous query accepted')

outer = groupby('AAB')
_, first = next(outer)
assert next(first) == 'A'
_, second = next(outer)
assert list(first) == []  # Advancing outer discarded the rest of its first run.
assert list(second) == ['B']
print('runs:', [key for key, _ in runs])
print('bad:', bad, 'fixed:', fixed, 'macro:', macro)
print('counts, noncontiguous-key guard, and shared-iterator checks passed')

The correct summaries are Q1:(1,2) and Q2:(1,1), giving an equally weighted query mean of 3/4. Giving each attempt equal weight instead produces 2/3. That is a different metric, not another repair. State the denominator after fixing grouping; selecting the larger result is not evidence of improvement. This example invokes neither a retriever nor a language model and establishes no gain in model accuracy.

Put ordering and counts in the evaluation interface

An evaluation interface should specify whether input is clustered by key or arbitrarily ordered, whether a row represents an attempt or a query, and what happens when a key reappears. Concurrent completion order, concatenated shards and a changed file sort can all change the runs. An adjacency-based implementation should not silently take responsibility for global aggregation.

If all key state cannot fit in memory, partitioning by key or external sorting may help organize aggregation. Each requires its own integrity and resource checks; this program implements neither. Before choosing groupby, sorting or a hash accumulator, establish when the pipeline can know that every record for a query has arrived.

Additional reading linked at backfill: why Python zip silently drops evaluation samples covers a separate alignment failure. Here the issue is how runs and overwrites change a summary after records arrive. 中文版. If these counterexamples and checks help you find missing counts in a summary, voluntary support is welcome.

The original article date is October 24, 2026 at 14:00 Beijing time; actual backfill is October 31, 2026. The fixed CPython documentation predates the slot. The example and both diagrams are original teaching material.

Support

If the explanations or examples helped you solve a problem or save time, you’re welcome to support this site with a voluntary tip.

WeChat support QR code; click to enlarge
WeChat
Alipay support QR code; click to enlarge
Alipay
Buy Me a Coffee; support this site
Buy Me a Coffee

Click a QR code to enlarge. More options: support page。

Share this Post

Leave a Comment

您的邮箱地址不会被公开。 必填项已用 * 标注

*
*