GitHub AI Scan: A Green Check Is Not Complete Coverage

黎 浩然/ 19 10 月, 2026/ 大语言模型/LARGELANGUAGEMODEL/LLM, 教程/TUTORIAL, 计算机/COMPUTER/ 0 comments

GitHub AI Scan findings are advisory, not an automatic merge gate. A passing pull-request check therefore does not establish that AI analysis completed without findings. Not applicable, queued, failed and completed with no findings need separate records.

Contents
  1. The release adds a detection signal
  2. No finding can conceal several different states
  3. Ten pull requests create three denominators
  4. Eight checks preserve missing versus empty results
  5. Keep check status and human decisions evidence-based
  6. Do not mistake workload differences for quality differences
  7. Check costs and handling capacity before adoption
  8. Leave unknown states visible
  9. Sources and backfill note

GitHub announced general availability of AI Scan for pull requests on October 15, 2026. The practical question here is how to add its signals without allowing one green status to hide missing coverage. Ten synthetic pull-request records separate eligibility, completion coverage and the no-finding proportion. 中文版.

The release adds a detection signal

The release announcement positions AI Scan as a complement to CodeQL for selected languages and frameworks it does not currently cover, including PHP, Shell/Bash and Dockerfiles. Findings appear in pull-request discussion and changed-file views with an AI indicator, and may include a proposed fix.

The same announcement states that findings do not fail the code-scanning check or block merges through ruleset requirements. Pull requests from forks and those created by Dependabot are excluded. These are the boundaries announced on October 15, rather than promises about every future version. No repository setting was enabled for this article.

The additional signal provides leads for the existing review process. It does not automatically replace semantic analysis, project tests or release responsibility. Decide who handles a finding and what evidence is required next, instead of treating the presence of an AI indicator as a complete security policy.

PR changes → advisory AI findings
Project review → confirm conditions and impact
Fix and regression → explicit merge decision
Original responsibility map: detection, human judgment and merge rules need separate evidence.

No finding can conceal several different states

The official documentation describes pull-request analysis rather than full-repository scans and notes possible false positives. Its scope cannot support a whole-repository absence-of-vulnerabilities conclusion. The state model below is an engineering proposal, not a claim that GitHub exposes identical API fields.

First, a pull request can be inapplicable. Second, it can be eligible without an available result, because completion is unconfirmed, waiting or failed. Third, analysis can complete with no findings. Fourth, completed analysis can report findings that still need human confirmation. Converting the first two states from missing values into empty lists invents evidence that analysis completed.

No findings is also weaker than no vulnerabilities. It describes one reported result for a recorded revision, analysis scope and execution conditions. Without an independent reference answer, those counts cannot establish a false-negative rate or recall. Accurate coverage bookkeeping is a prerequisite for evaluating detection, not detection quality itself.

Ten pull requests create three denominators

Suppose ten explicitly fictional pull requests include six eligible cases. Four complete, one is queued and one fails. Of the four completed results, three have no findings and one reports a placeholder teaching issue. The remaining four pull requests are inapplicable. No real repository, vulnerability or user information is involved.

Completion coverage among eligible pull requests is 4/6, or two-thirds. Relative to all pull requests, it is 4/10, or two-fifths. Both answer reasonable questions: how much expected work finished, and how much total change received this analysis. Selecting only the larger proportion would hide the denominator change.

The three no-finding results represent 3/4 of completed analyses. Treating all six missing-result records as clean would instead yield an incorrect 9/10. That ninety percent is not an optimistic estimate; it changes what the states mean. For a particular review, first ask whether a usable result exists before counting what it found.

All PRs 10 → eligible 6
Eligible 6 → complete 4: coverage 2/3
Complete 4 → no findings 3: proportion 3/4
Original synthetic ledger: distinct denominators answer distinct questions, without measuring real scan performance.

If an observation window contains no eligible pull requests, completion coverage has a zero denominator. Report it as not applicable with the count, rather than zero-percent or hundred-percent coverage. An earlier completed result should also not count as coverage of a changed head revision without evidence that it applies.

Eight checks preserve missing versus empty results

The standard-library example represents completed no-finding output with an empty list and missing output with None. Names and status values are teaching conventions, not a GitHub REST response. The script does not connect to GitHub, scan code or measure product detection quality.

# Synthetic coverage bookkeeping, not GitHub API or vulnerability scanning.
from fractions import Fraction as F
rows=[
    {"id":"a","eligible":True,"status":"complete","findings":[]},
    {"id":"b","eligible":True,"status":"complete","findings":[]},
    {"id":"c","eligible":True,"status":"complete","findings":[]},
    {"id":"d","eligible":True,"status":"complete","findings":["demo-1"]},
    {"id":"e","eligible":True,"status":"queued","findings":None},
    {"id":"f","eligible":True,"status":"failed","findings":None},
    *[{"id":i,"eligible":False,"status":"not_applicable","findings":None}
      for i in ["g","h","i","j"]],
]
eligible=[r for r in rows if r["eligible"]]
complete=[r for r in eligible if r["status"]=="complete"]
no_findings=[r for r in complete if r["findings"]==[]]
assert len(rows)==10
assert len(eligible)==6
assert len(complete)==4
assert len(no_findings)==3
assert F(len(complete),len(eligible))==F(2,3)
assert F(len(complete),len(rows))==F(2,5)
assert F(len(no_findings),len(complete))==F(3,4)
assert sum(r["findings"] is None for r in rows)==6
print("8 scan-state bookkeeping checks passed")

All eight assertions passed locally. They check total, eligible, completed and no-finding counts, three distinct proportions, and the six missing-result records. A production system must use completion evidence and revision identifiers actually available from its analysis jobs. Zero comments alone cannot establish completed analysis.

Keep check status and human decisions evidence-based

Because the product’s findings do not automatically block merging, a green code-scanning check must be interpreted according to its own rules. If a team requires human review for particular changes, define responsibility and evidence in that review process. Do not assume advisory output has become an enforced gate.

For a finding, first verify that it refers to the current pull-request revision rather than an earlier commit. Then inspect the relevant path, preconditions and consequences. A confirmed issue needs suitable regression evidence for its fix. A false-positive decision also needs a specific explanation; neither “the AI is wrong” nor “the AI found it” preserves a useful technical judgment.

A proposed fix remains a change to review. It can improve the reported area while changing compatibility or business behavior elsewhere. Read the actual diff and run relevant tests rather than merely checking whether the original comment disappeared. No exploit instructions or attack results are used as review evidence here.

Do not mistake workload differences for quality differences

Teams have different pull-request mixtures. Languages, change sizes, fork contributions and automated dependency updates can change the eligible set. Comparing findings per hundred pull requests may primarily compare workloads, without establishing that one detector is more accurate.

A more useful record stratifies applicable categories and retains completion state, available analysis configuration or version, disposition and human handling time. Begin with a fixed observation window to check record completeness. Evaluation against a reference set can follow if suitable evidence exists; without it, describe coverage and workload rather than recall.

This connects to the earlier ReviewBench discussion of comments and defect coverage. Finding count is not confirmed-defect count, and execution coverage is not vulnerability recall. Names and denominators must remain explicit when these measures share a report.

Check costs and handling capacity before adoption

The announcement says no separate Copilot license is required, but AI Scan consumes Code Security AI Credits. Availability also depends on repository ownership, visibility and the relevant security product. Check the effective account setup before adoption rather than interpreting the license statement as zero-cost operation.

No billing records were accessed and no feature was enabled for this article. It does not give an unverified per-scan price. Consider both scan usage and the work of reviewing findings. More suggestions cannot create engineering value by themselves if nobody confirms, reproduces or resolves them.

A bounded observation period can state which pull requests should qualify, who confirms completion, who handles findings, and how missing output is reported. Answering those four questions before broadening adoption reduces investment in a tool without an operational follow-through.

Leave unknown states visible

General availability adds another detection entry point, while merge decisions still require explicit evidence. Preserve unknown when analysis did not run or did not complete. Interpret completed no-finding output only within its actual scope.

If the state model, denominator calculations and example records help avoid confusing a green check with complete coverage, voluntary support through the section below is welcome. Their purpose is to make each review explain what was checked and what remains unknown.

Sources and backfill note

Discovered through Google News, then checked against the October 15 announcement and official documentation. Product facts used here are supported by the announcement before the planned slot; the ledger and diagrams are original. The article retains October 19, 2026 at 11:30 Beijing time as its planned date and was backfilled that evening. No real scanner or GitHub API ran.

Support

If the explanations or examples helped you solve a problem or save time, you’re welcome to support this site with a voluntary tip.

WeChat support QR code; click to enlarge
WeChat
Alipay support QR code; click to enlarge
Alipay
Buy Me a Coffee; support this site
Buy Me a Coffee

Click a QR code to enlarge. More options: support page。

Share this Post

Leave a Comment

您的邮箱地址不会被公开。 必填项已用 * 标注

*
*