If a detached tensor is appended to a list and the recorded numbers later change, inspect shared storage first. detach removes the current graph relationship; it does not automatically copy data. clone copies data but can still propagate gradients. A numerical record independent of later source mutations and disconnected from the original graph needs both requirements addressed.

For “What is the difference between detach and clone, and why is detach not a snapshot?”, two independent counterexamples separate storage mutation from stopped gradients. Python arrays below are not presented as a PyTorch execution, and a stopped-target objective is not differentiated as if its target still varied. 中文版.

Draw two relationships instead of inspecting one flag

The PyTorch 2.8 detach definition says the result leaves the current graph and does not require gradients, while sharing the original storage. In-place changes through either tensor can be visible through the other and may trigger correctness errors. A new Tensor object is not necessarily new underlying data.

The clone definition returns a copy while remaining differentiable, allowing gradients to flow to the input. Draw one connection for graph dependency and another for shared storage. detach addresses the former; clone addresses the latter. A requires_grad value alone does not establish that both connections have been removed.

This article concerns ordinary real-valued tensors, not every special layout, synchronization mechanism or serialization implementation. Renaming a variable or copying a Python container holding tensors likewise does not establish independent numerical data merely by increasing the object count.

A saved reference is not a saved historical value

Suppose a producer repeatedly reuses a buffer for intermediate results. A consumer stores a detached view or alias each step, then writes its records later. If the underlying buffer is overwritten, the consumer can observe newer values instead of values at the instant the reference was saved. Gradient recording and buffer reuse are independent conditions.

The Python storage analogy below starts source=[2,6], with shared referencing the same data and snapshot copying it. Changing the first source element to nine changes what shared reads. Writing twelve to the second element through shared changes source too. snapshot remains [2,6]. This executes the Python example’s alias behavior; the official definitions above support the PyTorch semantics.

Define ownership for logs, visualization caches and stored targets: should they follow the current buffer or freeze historical data? The latter requires independent storage or equivalent completed persistence, rather than merely stopping gradients. Real snapshots also need appropriate completion of asynchronous device operations. This standard-library example does not validate CUDA streams or cross-device copies.

Why a clone still propagates gradients

Copying values need not change their numerical function. Let y=3x and let z be a differentiable copy of y. Algebraically z remains 3x. With L=z², the chain rule gives dL/dx=2z×3, which is thirty-six at x=2. Allocating another storage object does not invalidate that chain rule.

Consequently, cloning to prevent a later overwrite and detaching to avoid training an upstream branch are not interchangeable fixes. One addresses data ownership, the other defines an optimization objective. Detaching an upstream branch that should train changes its gradient path. Conversely, cloning a teacher target can retain a reverse path that the design intended to exclude.

When both requirements apply, detach followed by clone explicitly disconnects the existing graph relationship and copies the data into independent storage. This does not make the result permanently read-only or incapable of participating in a different computation graph later. It describes the relationship being removed and the data being copied now.

Which derivative changes when a target is stopped?

Take x₀=2 and t=3x₀=6, with L=(x−t)². Treating t as fixed during this differentiation gives dL/dx=2(x−t), or −8 at x₀. If t instead varies as 3x, the function becomes (x−3x)²=4x² and its derivative at the same point is sixteen.

More precisely, fixing t takes a partial derivative with t as an independent parameter, while moving t takes the composite function’s total derivative. Both paths have loss sixteen at x₀, yet their gradient directions oppose each other. Equal forward values do not establish equal optimization behavior. Stopping a gradient is not merely repairing a numerical discrepancy. It declares that the target’s generating path does not contribute to this differentiation.

A finite-difference check can easily test the wrong function. To check the fixed-target derivative, evaluate both x₀+h and x₀−h with the same t=6. Recomputing t=3x for each perturbation instead evaluates the moving-target derivative. The program checks −8 and sixteen separately, preventing one from being used to reject the other’s valid result.

One update makes the distinction concrete. Starting from x=2 with the fixed-target gradient −8 and step size 0.1 gives x=2.8. Loss against the old target six falls from sixteen to 10.24. Recomputing the next target as 3x=8.4 instead gives 31.36. There is no contradiction: the two checks use different targets. Stopping a target can be a valid design without guaranteeing monotonically decreasing loss after targets are regenerated.

Ten checks separate storage from calculus

The standard-library program first checks an array alias and an independent copy, then applies central differences to the two scalar objectives. Its final derivative check covers the algebraic relation z=3x for a differentiable copy. PyTorch is not installed in the current environment, so no tensor autograd, model-training or device test was executed.

# Python storage analogy and finite differences, not a PyTorch runtime test.
from array import array
from math import isclose

source=array('d',[2.,6.])
shared=memoryview(source)
snapshot=array('d',source)
assert list(shared) == list(snapshot) == [2.,6.]
source[0]=9.
assert shared[0] == 9.
assert snapshot[0] == 2.
shared[1]=12.
assert source[1] == 12.
assert snapshot[1] == 6.

x0=2.; fixed_target=3*x0
frozen=lambda x:(x-fixed_target)**2
moving=lambda x:(x-3*x)**2

def derivative(f,x,h=1e-5):
    return (f(x+h)-f(x-h))/(2*h)

assert frozen(x0) == moving(x0) == 16.
assert isclose(derivative(frozen,x0),-8.,rel_tol=1e-8)
assert isclose(derivative(moving,x0),16.,rel_tol=1e-8)
# Independent storage alone does not alter the algebraic identity z=3*x.
cloned_expression=lambda x:(3*x)**2
assert isclose(derivative(cloned_expression,x0),36.,rel_tol=1e-8)
assert list(snapshot) == [2.,6.] and list(source) == [9.,12.]
print('10 storage and derivative checks passed')

This scalar finite-difference calculation is not a general gradient checker. A real model test must control randomness, differentiability assumptions and numerical precision before comparing its required gradient paths. The storage analogy also does not implement PyTorch’s version counters or in-place error detection. It demonstrates why a second object referencing the same data can change.

detach is not a universal remedy for in-place errors

The autograd mechanics notes explain in-place changes and version counters: backward can require values saved during forward, and modifying them affects correctness checks. A detached alias is therefore not a safe channel for changing source data without affecting differentiation.

When an in-place error occurs, identify which writer changed which storage and whether backward still needs the old value. Then decide whether copying, changing operation order or redesigning the computation is appropriate. Do not start by treating detach as a switch for suppressing the error. Avoiding a correctness check is not positive evidence that the intended derivative was computed.

The earlier view and reshape storage analysis supplies a related connection: shape operations can preserve aliases, and graph-disconnection operations can preserve aliases too. Names, shapes and gradient requirements individually fail to describe the entire storage lifetime.

Give recording and training separate acceptance criteria

For a log or cache, save a numerical record, mutate the producer buffer under controlled conditions, and check whether the record changes according to its ownership contract. For training, separately specify which parameters should receive gradients, whether a target branch is fixed, and the corresponding mathematical derivative. One test should not stand in for the other.

Record snapshot frequency and scope too. Independent copies consume storage, and copying every intermediate tensor is not automatically appropriate. A requirement for a few statistics can use a smaller record; reproducing complete numerical state requires explicitly preserving that data. No unmeasured memory or timing improvement is promised here.

If the alias counterexample and derivative comparisons help diagnose changing records or an incorrect gradient direction, voluntary support through the section below is welcome. The reusable result is a method for testing storage ownership separately from the optimization objective.

Sources and backfill note

detach · clone · Autograd mechanics

Checked against versioned PyTorch 2.8 definitions available before the planned slot. Examples, derivations and diagrams are original teaching material. Standard-library storage and finite-difference checks ran; PyTorch did not. This backfill keeps October 21 09:00 Beijing as its article date, with actual publication time recorded separately.