More AI code-review comments do not necessarily mean more independent defects found. Restating one issue three ways increases comment volume without expanding coverage. Evaluate whether comments are valid, which known issues remain missed, and how much human work is required to verify and handle the output.

GitHub’s ReviewBench provides public evaluation material for this question. Rather than repeat a leaderboard, this article uses pre-labeled synthetic findings to explain denominators and deduplication units. No review model is called, and the example does not reproduce the official evaluator. 中文版.

Establish what ReviewBench provides

Google News includes the GitHub introduction published October 5, 2026. It discusses evaluating code review on real pull requests, multi-source reference findings, and grounded and augmented metric families. It is a public benchmark announcement, not a model evaluation performed here or a peer-reviewed paper.

The project repository publishes PR material, reference findings and evaluation resources; its full set contains 219 PRs. Check corpus, judgment rules and versions before using results. Dataset size, agreement figures or a leaderboard position cannot be converted directly into reliability on another repository.

A, B, C and D below are fictional issue identifiers, not model names. They illustrate an auditable counting method. ReviewBench’s actual semantic matching, judgment and scoring rules must be checked against its implementation.

Five comments can cover only two issues

Suppose a reference set contains four known issues: A, B, C and D. Reviewer One emits five comments identifying A, A, B and two items already judged invalid, labeled noise. Under these synthetic labels, three comments refer to real issues, but only two distinct issues are covered. C and D remain missed.

The fraction of emitted comments judged valid is 3/5. Coverage of the four known issues is 2/4. The units differ: one counts comments, the other issues. Dividing three by four to claim 75% coverage would give a repeated finding credit as a nonexistent new discovery.

Now reviewer Two emits A, B and C plus five invalid comments: eight comments total. Known-issue coverage is 3/4, while comment validity is 3/8. Comment volume alone cannot select a winner. Reporting just one of the two fractions hides the trade-off in the other direction.

Reproduce counts with explicit labels

This code computes exact statistics from the assigned identifiers. The example author has already judged noise invalid. The code does not understand natural language or confirm defects. Its reference set and report lists are nonempty; an evaluation must separately define how empty output is reported rather than silently assigning an arbitrary precision.

# Synthetic, pre-labeled findings; not the ReviewBench evaluator.
from fractions import Fraction as F

gold = {"A", "B", "C", "D"}
review_a = ["A", "A", "B", "noise1", "noise2"]
review_b = ["A", "B", "C"] + [f"noise{i}" for i in range(5)]

def summarize(reports):
    matched_comments = sum(x in gold for x in reports)
    found = set(reports) & gold
    return matched_comments, F(matched_comments,len(reports)), F(len(found),len(gold))

a = summarize(review_a)
b = summarize(review_b)
assert len(review_a) == 5
assert a[0] == 3
assert a[1] == F(3,5)
assert a[2] == F(1,2)
assert len(set(review_a) & gold) == 2
assert len(review_b) == 8
assert b[1] == F(3,8)
assert b[2] == F(3,4)
print("Synthetic finding counts verified")

The local checks passed: reviewer One has comment validity 3/5 and known-issue coverage 1/2; reviewer Two has 3/8 and 3/4. No composite winner score is calculated because preferences over false alarms, missed issues and human handling cost have not been specified.

This function is not a ReviewBench-compatible scorer. Real comments can point to different lines while describing one root cause. Similar wording can also describe separate defects. Identifier equality is sufficient for this teaching set and insufficient for actual review judgment.

Deduplicate around root cause and repair

If one bounds error is described at both a function entry and its caller, deciding whether the reports duplicate each other requires examining root cause, whether one repair resolves both, and whether each contributes independent necessary information. Line-number deduplication can miss repeats; keyword deduplication can merge distinct issues.

An inspectable annotation can retain issue ID, triggering condition, affected behavior, evidence location, proposed repair scope and relationships to other comments. Keep disputed cases rather than forcing every finding immediately into certain true or certain false.

Longer wording should not earn additional coverage credit. More explanation may reduce comprehension effort, but it does not automatically discover another defect. Validity, coverage and explanatory clarity can be observed separately instead of forced into one count.

Unmatched is not automatically false

A real PR’s reference findings are difficult to make exhaustive. An unmatched comment might be noise or a valid issue omitted from the initial labels. Preserve the distinction between unmatched and confirmed invalid; the matcher should not silently replace substantive judgment.

This is a useful question when reading grounded and augmented metrics: a fixed reference set and newly discovered findings serve different purposes. Consult the official method for its actual expansion and judgment rules. The synthetic code here does not implement assessment of new issues.

If each system’s new findings expand its own denominator, cross-system comparison raises another accounting question. State whether the reference set is fixed, how discoveries are confirmed, when versions change, and whether prior results can be recomputed under the same labels.

An average does not replace category checks

A reviewer may catch simple missing tests while performing poorly on concurrency state or recovery paths. Aggregating everything into one average can let numerous easy cases obscure fewer consequential misses. Examine categories relevant to the code you maintain.

Record PR size, language and available context too. Reviewing a diff alone differs from reading related implementation and tests. Model versions, prompts and tool configuration can change outcomes. Scores without these conditions are hard to interpret.

Use a public ranking to identify candidates for further inspection, rather than treating first place as a guarantee for every project. GitHub develops both the benchmark and a review product. That context is worth knowing, but does not alone establish that the data is correct or incorrect; inspect published methods and reproducible evidence.

Record the human work alongside findings

Invalid comments require reading and dismissal; valid ones still require verification and repair. An authorized real trial can record valid issues, duplicates, disputed cases, misses and the handling process for each PR. Clicking or replying to a comment does not establish that an issue was fixed.

Without timing data, do not multiply comment count by an invented number of minutes and claim labor savings. Leave the result unknown until comparable tasks and completion standards provide evidence. No real-repository trial was performed here, so no productivity increase is reported.

The earlier ambient-AI task acceptance article examines execution evidence. Review output also needs stable counting units and inspectable labels. Rich-looking prose is insufficient evidence of useful review.

Agree on the denominator before reading the ranking

To compare reviewers, first fix PRs and versions, reference findings, deduplication rules and relevant categories. Then inspect coverage and noise. Retaining failures and disputed cases helps assess workflow suitability more than a single overall-score screenshot.

If the reproducible counts and annotation method help you evaluate AI code review more clearly, voluntary support through the section below is welcome.

Sources and backfill note

Discovered through Google News and checked against the October 5 GitHub introduction and official repository. Both figures and synthetic reviewers are original teaching material. Local counting checks passed; no official benchmark or real product comparison ran. The article retains October 19, 2026 at 16:00 Beijing time as its planned date and was backfilled later that day.