LoRA rank controls the width of a low-rank branch, while alpha participates in its output scaling. Increasing rank with fixed alpha also reduces the coefficient under the standard alpha/r rule. A rank experiment can therefore mix capacity with update scale. rsLoRA changes the denominator to sqrt(r), which must be recorded as another experimental condition.

A reproducible first-step update below shows what changes together and how to design an interpretable comparison. 阅读中文版.

Write the coefficient that enters the forward pass

Using column vectors, a linear adapter gives y=Wx+sBAx. A maps inputs to rank r and B maps back to the output dimension. Standard LoRA uses s=alpha/r; rsLoRA uses s=alpha/sqrt(r). Alpha is not itself the learning rate. The coefficient multiplying the branch is s.

The PEFT 0.21.0 configuration reference specifies these rules for use_rslora. With layer-specific rank or alpha settings, inspect effective values per layer instead of only global defaults. The site’s LoRA parameter-count article covers matrix sizes; here the question is how scaling enters an update.

The same alpha does not mean the same scale

With alpha=8 and r=16, standard scaling is 0.5 and rsLoRA scaling is 2. Holding A, B, and input fixed makes the latter branch output four times larger. In general, their coefficient ratio at equal rank and alpha is sqrt(r).

This compares one forward pass with fixed matrices, not matrices learned by two training runs. Enabling use_rslora while retaining other settings can change initial update scale. Treat it as an experimental condition, not a switch guaranteed to improve an evaluation score.

Zero initial output can still permit learning

A common initialization sets B to zero and initializes A randomly, making BAx zero. The initial forward pass then matches the original linear layer. PEFT describes this default no-op initialization; other initialization strategies can differ and need inspection.

In this ordinary linear branch, the gradient for A contains B, so it is zero initially when B=0. The gradient for B contains A and can be nonzero given suitable input and upstream gradient. Zero adapter output therefore does not imply zero gradients for both matrices.

Calculate one complete SGD step

Construct scalar input x=1, base weight W=0, target 1, and loss L=½(y−1)². Let A contain r alternating ±1 entries and initialize B to zero. Use ordinary SGD with learning rate eta. The initial error is −1, and the gradient for entry bᵢ is −s aᵢ.

After updating, bᵢ=eta s aᵢ, giving branch output sΣbᵢaᵢ=eta s²Σaᵢ²=eta s²r. Standard scaling yields eta alpha²/r, while rsLoRA yields eta alpha². This rank dependence follows from a specified construction, not an unconditional law for every stage of real-model training.

# Scalar linear teaching model. No PEFT, neural model, or GPU is run.
import math

def scale(alpha, rank, rs=False):
    if not isinstance(rank,int) or isinstance(rank,bool) or rank<=0:
        raise ValueError("rank must be a positive integer")
    return alpha/(math.sqrt(rank) if rs else rank)

def one_step(rank, alpha=8., eta=0.01, rs=False):
    a = [1. if i%2==0 else -1. for i in range(rank)]
    b = [0.]*rank
    x, base_w, target = 1., 0., 1.
    s = scale(alpha,rank,rs)
    pred = base_w*x+s*sum(ai*bi for ai,bi in zip(a,b))*x
    error = pred-target  # loss = 0.5 * error**2
    grad_b = [error*s*ai*x for ai in a]
    grad_a = [error*s*bi*x for bi in b]
    assert all(g==0 for g in grad_a)
    assert any(g!=0 for g in grad_b)
    b = [bi-eta*g for bi,g in zip(b,grad_b)]
    output = base_w*x+s*sum(ai*bi for ai,bi in zip(a,b))*x
    merged_w = base_w+s*sum(ai*bi for ai,bi in zip(a,b))
    assert math.isclose(output,merged_w*x,abs_tol=1e-12)
    return output

for r in [4,16,64]:
    standard = one_step(r)
    stabilized = one_step(r,rs=True)
    assert math.isclose(standard,0.64/r,abs_tol=1e-12)
    assert math.isclose(stabilized,0.64,abs_tol=1e-12)
    assert math.isclose(scale(8,r,True)/scale(8,r),math.sqrt(r))
    print(r,round(standard,4),round(stabilized,4))
assert one_step(16,eta=0.)==0.
try:
    scale(8,0)
except ValueError:
    pass
else:
    raise AssertionError("invalid rank accepted")

The code was executed locally. At alpha=8 and eta=0.01, standard outputs for ranks 4,16,64 are 0.16,0.04,0.01; rsLoRA gives 0.64 for each. Checks also cover initial gradient differences, merged forward equality, zero learning rate, and invalid rank. No PEFT run or model fine-tuning was performed.

What this example establishes—and what it does not

Scaling enters twice in this update: once in the gradient and again in the post-update forward pass. Describing a fourfold coefficient change alone does not describe the output change after SGD. At fixed rank in this construction, the ratio of first-step outputs is r.

The rsLoRA source, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, was posted as an arXiv preprint on November 28, 2023. It studies rank stability under particular initialization, moment, and asymptotic conditions. The scalar example is not a complete proof of that theorem.

Its alternating A entries make Σaᵢ² explicit; they are not a real model’s weight distribution. Data, nonlinear layers, optimizer choice, precision, and later changes to both matrices affect training. The plotted 0.64 is not a prediction of task loss or accuracy.

Can a learning-rate change replace scaling?

For this first step with ordinary SGD and other quantities fixed, eta s² controls the output change. Different eta and s can therefore be chosen to match this first output. That restricted equivalence does not establish an identical training trajectory.

Later steps update both matrices; momentum and adaptive optimization add historical state, and other trainable modules may use separate learning rates. Saying scaling is completely equivalent to changing learning rate omits these conditions. Conversely, changing scaling does not remove the need to reconsider optimizer settings.

A fixed coefficient is not a fixed first update

One can increase alpha with rank so standard LoRA keeps alpha/r=c. Under rsLoRA, alpha=c sqrt(r) also keeps s=c. These choices fix the forward multiplier, but the construction here still gives first-step output eta c²r because the sum of squared A entries is r.

Holding alpha fixed, holding s fixed, and holding the first updated output fixed are three different comparisons. State which one an experiment controls instead of saying only that scaling was kept consistent. A hyperparameter table needs initialization and update details too; one ratio does not describe every change.

Make a rank comparison interpretable

Fix data splits, target modules, training-token budget, batch conditions, and optimizer. Decide whether the comparison concerns capacity or a complete training recipe. If only r changes while alpha is retained, s changes under standard LoRA too; record both rather than attributing everything to rank.

For each run, retain r, alpha, effective s, trainable parameter count, initialization, learning rate, and budget. Choose metrics from task requirements. Validation loss and labeled task accuracy answer different questions; lower training loss does not by itself establish improved generalization or output quality.

Record resource conditions as well. Higher rank generally adds adapter parameters and training costs. Merged inference and inference with separate adapter branches are different paths. Do not transfer one path’s cost conclusion to the other. No unmeasured memory or throughput table is provided here.

Save the scaling contract with the matrices

Loading the same A and B under another s changes deltaW=sBA. Retaining matrix files while losing rank, alpha, use_rslora, or base-model version undermines reproduction. Layer-specific settings should also preserve which modules actually matched them.

Before merging or exporting, compare unmerged and merged outputs on small non-sensitive inputs. Hold precision and inference mode fixed, with a tolerance justified by actual numerical conditions. The scalar code checks the corresponding equality, but does not validate a quantized checkpoint or deployment framework.

Choose alpha after defining the comparison

There is no model-, task-, and budget-independent best alpha. Check effective scaling and initialization first, then use controlled small experiments to inspect updates and validation behavior before investing in higher ranks. The result is conditional evidence, rather than a certainty copied from a hyperparameter table.

If this derivation and recording method help expose a hidden scale change in a rank experiment, voluntary support through the section below is welcome.

Sources and backfill note

The PEFT 0.21.0 LoRA reference and rsLoRA preprint. Both diagrams, the scalar construction, and its numerical values are original teaching content, without model-training results. Backfilled on October 16; the article date retains the planned 09:00 Beijing slot.