NTT Traceability: Why Passing Tests Can Miss Requirements

黎 浩然/ 24 10 月, 2026/ 计算机/COMPUTER/ 0 comments

A green test suite can mean that the tests and the implementation share the same mistake. NTT’s October 23, 2026 announcement about software-asset traceability addresses inconsistencies across requirements, design, code and tests. For developers, the practical question comes before generating more tests: where will their expected results come from?

Contents
  1. Checking relationships between software artifacts
  2. Two losses of information, two passing tests
  3. An oracle needs a version and room for disagreement
  4. How a 12-cell decision table finds the differences
  5. A counterexample still needs a reviewed repair
  6. Make the basis of a test a deliverable

Checking relationships between software artifacts

The original NTT announcement describes organizing artifacts by function, reconstructing relationships between upstream and downstream work, and adding missing testing perspectives. The two checks serve different purposes: cross-artifact alignment looks for inconsistent descriptions, while testing-perspective analysis looks for conditions that remain unverified and can organize additional cases using decision tables. Retrieving more relevant passages is not, by itself, equivalent to performing either check. In a preliminary company evaluation on the Macchinetta sample application ATRS, the approach found 33 of 41 deliberately embedded design-document errors, compared with 11 for the conventional RAG baseline. In a separate set of six planted code bugs, it found four; the company also reported three additional latent bugs.

These are company-reported results from a particular example, not a general detection rate. Document errors, planted code bugs and additional discoveries are separate populations. Combining them into one accuracy figure would obscure what was tested. The example below does not reproduce NTT’s system. It is an original, deliberately small model of what a traceability review needs to follow.

Two losses of information, two passing tests

Suppose requirement R1 says cancellation is allowed only when an order is paid, has not shipped, and the request arrives between zero and 30 minutes after payment, including minute 30. This is a fictional teaching contract, not a description of a merchant’s policy. The inclusive boundary is explicit so that an ambiguous phrase such as “within 30 minutes” does not quietly become an unapproved assumption.

Design D1 retains the unshipped condition but mistakenly changes the time limit to less than 30. Implementation C1 then drops the shipping condition and checks only payment and a time in the range 0≤minutes<30. The existing tests cover a paid, unshipped order at minute 10 and an unpaid, unshipped order at minute 10. Both pass. Nevertheless, a boundary and a business condition have disappeared along the way.

R1 requirement
Paid AND unshipped
0≤minutes≤30
↓ D1 design
Keeps unshipped
Changes limit to <30
↓ C1 implementation
Keeps <30
Drops unshipped
Tests T1 and T2
Only ordinary minute-10 paths
Both pass
Figure 1. Original artifact chain. R1→D1 loses the inclusive boundary; D1→C1 loses the shipping condition. A relationship arrow does not prove that the downstream artifact is correct.

Traceability supplies a useful lead: where did a condition first change? Comparing code with tests generated from that code may reveal no disagreement. Including the requirement and design brings a different source of intent into the review. A functional unit should also preserve the conditions, boundaries and exceptions needed to review cancellation together, rather than merely fit a fixed character count. This is an engineering explanation, not an implementation of NTT’s internal algorithm.

An oracle needs a version and room for disagreement

A test oracle supplies the expected behavior for an input. Here, required comes directly from the explicit R1 contract, while implementation represents intentionally defective code. Keeping them separate makes the comparison meaningful. If a tool reads implementation and copies its current behavior into every expected result, adding tests can preserve the defect more thoroughly.

This does not make documentation infallible. A requirement may have been superseded by an approved change; a design may omit that change; different tenants or versions may intentionally use different policies. After finding a difference, establish the versions, applicability and approval basis of the artifacts before deciding whether to change the code, design or requirement. A link to a requirement ID establishes a relationship, not the truth of its contents.

A compact review trail can connect the specific R1 condition, the relevant D1 passage, the C1 branch and a test that exercises the condition. The reviewer follows the meaning of that chain, rather than checking that all four identifiers exist. If the inclusive boundary remains disputed, preserve it as an unresolved question. Silently choosing an interpretation and generating passing tests would hide the business disagreement inside the test suite.

How a 12-cell decision table finds the differences

Take two payment states, two shipping states and three times: 29, 30 and 31 minutes. Their Cartesian product has 2×2×3=12 cases. The times sit before, on and after the boundary. Compared with the two ordinary-path tests, this small table deliberately covers combinations of conditions. It does not exhaust every possible input.

Existing checks
2 ordinary paths
✓ 2 / 2
Expanded cases
2 payment × 2 shipping states
× minutes 29, 30, 31
Difference A: unshipped, 30
Required: allow → actual: reject
Lost boundary
Difference B: shipped, 29
Required: reject → actual: allow
Lost shipping condition
Figure 2. Original coverage expansion and executed results. Two mismatches appear in 12 cases. These counts describe only this teaching program.

Save the following as traceability_example.py and run python3 traceability_example.py without -O, which would disable assertions. This standard-library program was executed with Python 3.15.0. Each output tuple gives payment state, shipping state and elapsed minutes, in that order.

from itertools import product


def required(paid, shipped, minutes):
    # R1: the agreed fictional contract, including minute 30.
    return paid and not shipped and 0 <= minutes <= 30


def implementation(paid, shipped, minutes):
    # C1: intentionally defective implementation.
    return paid and 0 <= minutes < 30


old_cases = [
    (True, False, 10, True),
    (False, False, 10, False),
]
assert all(
    implementation(paid, shipped, minutes) == expected
    for paid, shipped, minutes, expected in old_cases
)

cases = list(product([False, True], [False, True], [29, 30, 31]))
mismatches = [
    (paid, shipped, minutes)
    for paid, shipped, minutes in cases
    if implementation(paid, shipped, minutes)
    != required(paid, shipped, minutes)
]
assert len(cases) == 12
assert set(mismatches) == {(True, True, 29), (True, False, 30)}
print({
    "existing_tests_passed": len(old_cases),
    "decision_table_cases": len(cases),
    "mismatches": mismatches,
})

The program confirms that both old tests pass and that the 12-case table finds two differences. It invokes no language model and measures no NTT tool. The reusable result is the concrete counterexample: an unshipped order at minute 30 is incorrectly rejected, and a shipped order at minute 29 is incorrectly allowed. Those inputs are more useful for locating a repair than an unsupported claim that “coverage improved.”

A counterexample still needs a reviewed repair

These differences guide a repair, but do not approve one. First confirm that R1 still applies. Then restore the inclusive time boundary and the unshipped condition separately. Retain both counterexamples as regression checks and rerun the original ordinary paths, so that fixing one condition does not damage another. If the code is repaired while D1 remains wrong, the design also needs attention: future code generation could otherwise reintroduce the same defect.

The purpose of a decision table is not to enumerate as much as possible. It is to select boundaries and combinations that can change the result. The values 29, 30 and 31 expose these particular defects, but do not check negative or noninteger inputs, mixed time units, state changes after payment, or a concurrent shipment between checking eligibility and committing cancellation. A real interface that admits such cases needs the corresponding contract and verification. Passing this table cannot establish the correctness of a complete cancellation service.

A team introducing automated analysis could begin with one well-bounded feature and record the artifact versions, reproducible input, confirmed outcome and human disposition for each finding. Evaluate false alarms, missed defects and review effort, rather than counting generated tests alone. Also count an inconsistency finding separately from a confirmed defect. A finding may indicate an obsolete document or an undecided rule; establish its basis before counting a repair as effective. NTT’s sample results motivate further investigation of cross-artifact checking; they do not establish its return for every project, programming language or codebase size.

Make the basis of a test a deliverable

The most useful question to take from this announcement is not a detection percentage: which confirmed rule produced this expected value? When the answer identifies a version, condition and counterexample, automatic test generation has a chance to add missing judgments instead of amplifying the current implementation’s behavior.

Additional reading linked at backfill: fresh context in EGM design review examines another review perspective. This article examines whether the basis for a judgment is independent of the implementation being checked. 中文版.

The original article date is October 24, 2026 at 11:30 Beijing time; actual backfill is October 30, 2026. The news source was public on October 23. The diagrams and cancellation example are original teaching material. If the condition-tracing and counterexample method helps you inspect your own tests, voluntary support is welcome.

Support

If the explanations or examples helped you solve a problem or save time, you’re welcome to support this site with a voluntary tip.

WeChat support QR code; click to enlarge
WeChat
Alipay support QR code; click to enlarge
Alipay
Buy Me a Coffee; support this site
Buy Me a Coffee

Click a QR code to enlarge. More options: support page。

Share this Post

Leave a Comment

您的邮箱地址不会被公开。 必填项已用 * 标注

*
*