A SwiGLU feed-forward layer multiplies two linear projections elementwise: one passes through SiLU, the other supplies values, and a down projection returns the result to the model dimension. It adds a matrix, but a narrower intermediate width can keep the parameter budget close to an ordinary FFN. Matrix count alone does not determine cost.

One activation path becomes two branches

A bias-free ordinary FFN can be written y=φ(xW₁)W₂. Using row vectors, x has d components, W₁ has shape d×m and W₂ has shape m×d. The activation φ operates on each intermediate component, with width m.

Bias-free SwiGLU instead computes y=[SiLU(xWg)⊙(xWv)]Wo. Both Wg and Wv have shape d×h, and Wo has shape h×d. The symbol ⊙ means elementwise multiplication, not matrix multiplication. The branches must produce matching h-dimensional vectors.

The gate is not a zero-to-one switch

SiLU(t)=t·σ(t), where σ is sigmoid; this uses Swish with β=1. Sigmoid lies between zero and one, but multiplying it by t removes that range restriction from SiLU. Negative inputs can give negative outputs, and positive inputs can give values above one.

Gating describes input-dependent modulation, not a promise that each gate behaves like a switch or probability. The candidates are not normalized to sum to one, so this is not softmax. Two learned projections followed by multiplication distinguish this structure from a single activation path.

Noam Shazeer’s GLU Variants Improve Transformer was first submitted to arXiv in February 2020 and compares gated FFN variants. This tutorial uses the original manuscript for the architecture. It does not turn its preprint status into a verified journal-review claim, or extend the author’s particular experimental improvements to every model.

Compare widths as well as matrix counts

Without biases, an ordinary FFN has 2dm parameters and SwiGLU has 3dh. Equal parameter budgets imply h=2m/3. If m=4d, this gives h≈8d/3. Actual implementations may adjust the width for hardware-friendly alignment.

For an integer teaching example, d=6 and m=24 give 288 matrix parameters in the ordinary layer. With h=16, the gated layer also has 3×6×16=288. This is neither a real model configuration nor the total parameter count of a Transformer.

The dense-projection multiply-accumulate count per token scales approximately with 3dh, versus 2dm for the ordinary layer. Equal parameter budgets do not establish equal wall-clock speed: activation, elementwise multiplication, intermediates and kernels also matter. No GPU latency measurements or speedup claims are supplied.

Trace a two-dimensional input

import math

def silu(t):
    if t >= 0:
        sigmoid = 1.0 / (1.0 + math.exp(-t))
    else:
        e = math.exp(t)
        sigmoid = e / (1.0 + e)
    return t * sigmoid

def row_mm(x, matrix):
    if not matrix or len(x) != len(matrix):
        raise ValueError("Input dimension mismatch")
    width = len(matrix[0])
    if width == 0 or any(len(row) != width for row in matrix):
        raise ValueError("Expected rectangular nonempty matrix")
    return [sum(x[i] * matrix[i][j] for i in range(len(x)))
            for j in range(width)]

def swiglu(x, gate, value, down):
    g = row_mm(x, gate)
    v = row_mm(x, value)
    if len(g) != len(v):
        raise ValueError("Branch widths differ")
    hidden = [silu(a) * b for a, b in zip(g, v)]
    return row_mm(hidden, down)

identity = [[1.0, 0.0], [0.0, 1.0]]
value = [[1.0, 0.0], [0.0, -1.0]]
y = swiglu([1.0, 2.0], identity, value, identity)
assert math.isclose(y[0], silu(1.0))
assert math.isclose(y[1], -2.0 * silu(2.0))
assert swiglu([0., 0.], identity, value, identity) == [0., 0.]
assert silu(-1.0) < 0 and silu(2.0) > 1
assert math.isfinite(silu(-1000.0))
try:
    row_mm([1.], identity)
except ValueError:
    pass
else:
    raise AssertionError("Dimension mismatch accepted")
assert 2 * 6 * 24 == 3 * 6 * 16 == 288
print([round(a, 6) for a in y])

This original Python teaching implementation uses no training framework. For x=[1,2], the gate and down matrices are identities, while the value branch negates the second component. The output is approximately [0.731059,-3.523188]. The negative output follows the formula even though it conflicts with a probability-switch intuition.

The code was executed, checking the hand-derived result, zero input, SiLU’s range, sigmoid evaluation at a large negative input, and a dimension mismatch. The parameter-count example also passed an assertion. It processes one token only, without batches, backpropagation, biases or a residual connection.

Know which layer handles token interaction

An FFN typically applies the same weights independently at each sequence position. SwiGLU modulates features at that position; it does not itself query another token. Keep that role separate from cross-position interaction through attention.

When reading a configuration or source code, identify the actual gate, up and down shapes, then check intermediate width, biases, activation and residual placement. Different naming conventions do not change the row-vector derivation, although framework weight storage may use a transpose.

Replacing an activation structure affects shapes and training compatibility. Renaming a layer in an existing checkpoint does not obtain the paper’s results. This article gives a checkable explanation of architecture and budget; quality, throughput and memory tradeoffs still require measurements with specified models, precision and hardware.

阅读中文版

Source

Noam Shazeer — GLU Variants Improve Transformer, arXiv v1, February 12, 2020