Use expand when a size-one dimension only needs to be read as a larger shape. Consider repeat or an explicit copy when actual duplicated storage is required, particularly before editing each copy separately. Identical printed arrays do not establish identical storage. There is also a separate training question: copying data does not automatically create independently learnable model parameters.

A 1×3 vector expanded to 4×3 is enough to inspect addresses, writes, derivatives and downstream memory. The aim is to decide between expand and repeat from the program’s requirements, rather than nominate one universally faster expression. 中文版.

Four rows can refer to only three stored values

The PyTorch 2.8 expand documentation describes a view that uses a zero stride when enlarging a size-one dimension. Starting from x=[[1,2,3]], shape (4,3) exposes four logical rows, but every row can still read the same three stored values.

For a contiguous float64 source, adjacent columns are eight bytes apart. An expanded element has an address offset of i×0+j×8: changing row i does not move the address, while column j selects the value. Offsets here are measured in bytes, as are NumPy strides. PyTorch stride values normally count elements instead. The same layout can therefore be recorded as (0,8) bytes or (0,1) elements. Normalize the units before comparing logs from the two libraries.

The operation does not allocate twelve values and compress them afterward. It answers multiple reads using the same original locations from the beginning.

This explains the restriction as well. An existing dimension of size two cannot become four by this zero-stride mechanism while preserving two distinct values as ordinary expansion. Explicit repetition defines another operation. Write down both shapes and compare aligned dimensions from the trailing end before drawing conclusions from a printed matrix. The distinction matters when a tensor contains real variation along an axis that was mistakenly assumed to be shared.

Copied data does not imply independent parameters

The repeat documentation states that data is copied. It also distinguishes the operation from numpy.repeat: its behavior is closer to NumPy tile. The executable example therefore uses tile to construct equal values in separate storage, without assuming that identically named functions across libraries have identical interfaces.

Suppose both programs start from the same variable x, produce four numerical copies and compute L=Σᵢⱼwᵢⱼyᵢⱼ. Whether the intermediate values share storage or have been copied, the function of x remains yᵢⱼ=xⱼ. Consequently, ∂L/∂xⱼ=Σᵢwᵢⱼ. With weight rows (1,2,3), (4,5,6), (7,8,9) and (10,11,12), the derivative with respect to x is (22,26,30).

Differentiation asks how many uses change when that same input variable changes. Storage analysis asks where the current values reside. Allocating another buffer does not by itself remove the dependency on the input. In particular, four copied rows should not be mistaken for four independently learnable parameters. If the intended model needs independent parameters, express that choice in its parameterization rather than infer it from a copying operation.

The derivative here is a chain-rule derivation checked by finite differences of two numerical programs. It is not a PyTorch automatic-differentiation execution. Framework questions about differentiability, leaf tensors or optimizer registration need their corresponding semantics. The earlier detach and clone comparison separately examines storage relationships and gradient boundaries.

Separate storage before editing individual copies

If every row refers to the same location, changing only row zero, column zero may not represent an independent-copy update at all. The expand documentation warns about in-place operations on overlapping storage and recommends cloning before writing. Correctness should not depend on whether a particular vectorized mutation happens to raise an error or happens to produce plausible output.

The address experiment uses NumPy broadcast_to, which returns a read-only broadcast view. Read-only is the protection policy of that NumPy interface; it is not a statement that PyTorch expand has the same policy. Modifying the source provides a clearer aliasing check: all four view rows observe the new source value, whereas the earlier tiled copy keeps its previous value.

When independent edits are needed, copy first and modify one position afterward. In the program, c=y.copy() followed by changing c[0,0] leaves both the source and the other rows of c unchanged. That demonstrates independent storage for this copy. It does not demonstrate that a differentiation path has been detached. Whether storage is separate and whether dependency information is retained remain separate questions.

A cheap input view does not make the whole chain allocation-free

A logical 4×3 float64 array contains twelve elements, or 96 bytes at eight bytes each. The source holds only 24 bytes. A broadcast view still reports 96 logical bytes through nbytes. That figure does not prove that a fresh 96-byte data buffer was allocated. Conversely, the source’s 24 bytes do not prove that the complete expression needs only 24 bytes.

The out=y+ones expression in the program creates a new 4×3 result with 96 bytes of element data and no shared storage with x. More generally, if a later operation produces a dense result, avoiding an input copy saves only that input copy. Output data, temporary buffers and anything saved for backward require their own accounting.

For an explicit budget calculation, take (B,T,D)=(32,2048,4096) and two-byte elements. A full dense array contains 536,870,912 bytes, or 512 MiB; a (1,1,D) source contains 8,192 bytes. Expanding the input can avoid materializing that 512 MiB input copy. It cannot remove the budget for a dense output of the same shape when the following operation requires one. This is arithmetic, not a measured GPU allocation, and excludes other arrays, caching and overhead.

A subsequent reshape also should not be assumed to preserve zero-copy storage solely because of its name. Compatibility between shape and strides matters. The existing view and reshape storage discussion covers that stage. The focus here is reuse of an input and its consequences for mutation and derivatives.

Check storage, values and derivatives separately

The program actually runs NumPy arrays. It checks matching shape and values, zero row stride, view/source aliasing, copy/source independence, the read-only flag, propagation of a source edit, an independent edit after copying, logical byte counts, a downstream allocation, an invalid expansion, and finite differences for both numerical paths. The fourteen assertions are tied to these distinctions.

# NumPy storage experiment and explicit derivatives, not PyTorch autograd.
import numpy as np
x=np.array([[1.,2.,3.]])
y=np.broadcast_to(x,(4,3))
r=np.tile(x,(4,1))
assert y.shape==r.shape==(4,3)
assert np.array_equal(y,r)
assert y.strides[0]==0
assert np.shares_memory(x,y)
assert not np.shares_memory(x,r)
assert not y.flags.writeable  # NumPy policy; not a PyTorch guarantee.
x[0,0]=9.
assert np.all(y[:,0]==9.) and np.all(r[:,0]==1.)
c=y.copy(); c[0,0]=-1.
assert x[0,0]==9. and c[1,0]==9.
assert y.nbytes==r.nbytes==96 and x.nbytes==24
out=y+np.ones((4,3))
assert out.nbytes==96 and not np.shares_memory(x,out)
try:
    np.broadcast_to(np.ones((2,3)),(4,3))
except ValueError:
    pass
else:
    raise AssertionError('A non-singleton axis cannot expand this way')
x=np.array([[1.,2.,3.]])
w=np.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.],[10.,11.,12.]])
expected=w.sum(axis=0,keepdims=True)
assert expected.tolist()==[[22.,26.,30.]]
def loss(z,kind):
    a=np.broadcast_to(z,(4,3)) if kind=='view' else np.tile(z,(4,1))
    return float((a*w).sum())
for kind in ('view','copy'):
    numerical=np.empty_like(x)
    for j in range(3):
        a=x.copy(); b=x.copy(); a[0,j]+=1e-5; b[0,j]-=1e-5
        numerical[0,j]=(loss(a,kind)-loss(b,kind))/(2e-5)
    assert np.allclose(numerical,expected,rtol=1e-8,atol=1e-8)
print('14 storage and derivative checks passed; NumPy',np.__version__)
print('source bytes',x.nbytes,'logical broadcast bytes',y.nbytes)

Each finite-difference calculation perturbs one column of x while keeping weights and target shape fixed. Both programs define the same scalar objective, so their numerical derivatives should match the manually summed (22,26,30). PyTorch is unavailable in the environment. The experiment does not execute Tensor.expand, Tensor.repeat, backward or CUDA performance measurements.

Keep the reduction fixed as well. Replacing the weighted sum over the 4×3 array with a mean over all twelve entries divides the derivative by twelve, producing (11/6,13/6,5/2). The difference comes from the objective, rather than an expand-specific effect that mysteriously enlarges gradients. Use count, weights, sum and mean all belong in the derivation.

Choose from the intended use

For a shared bias or feature read at multiple positions, confirm that the broadcast axes really have size one and inspect whether the consuming operation accepts that layout. Many broadcasting operations do not require an explicit expand call beforehand. If the output copies need independent edits, budget for independent storage. If every copy must learn independently, revisit whether the model actually shares its parameters.

During debugging, retain three kinds of evidence: input/output shapes and strides; storage sharing among the source, copies and downstream results; and gradients or finite differences for a fixed loss. For performance questions, measure the complete forward and backward path on the actual device, including peak memory. No universal speed ranking between expand and repeat is established by this example.

If the address derivation and fourteen checks help distinguish avoided copies, independent writes and shared-variable derivatives, voluntary support through the section below is welcome. Adapt the example to your own shapes, verify its meaning first and compare performance afterward.

Sources and validation scope

PyTorch 2.8 expand · PyTorch 2.8 repeat · PyTorch broadcasting · PyTorch reshape · NumPy 2.3 broadcast_to

Checked against versioned PyTorch 2.8 documentation, with broadcasting notes last updated January 31, 2021. This engineering explanation is not presented as a new product announcement. The original example ran NumPy, not PyTorch or device benchmarks. Planned article date: October 22, 2026 at 23:30 Beijing; scheduling and public release are verified separately.