Manufactured Rank Is a Spectator: A Causal Test of Nonlinear Low-Rank Adapters in Task Adaptation

Abstract

A growing family of parameter-efficient adapters proposes to improve on LoRA by inserting an element-wise nonlinearity between the low-rank factors, manufacturing rank beyond the nominal ceiling without adding parameters. The premise is correlational: the nonlinearity provably raises the realized update’s stable rank, and higher rank co-occurs with higher accuracy. We test it causally and find the manufactured rank is a spectator. A taxonomy first narrows the target: some methods advertised under “nonlinearity as rank” do not raise the realized rank at all (genLoRA proves $\mathrm{rank}(\Delta W)\le r$ by its own construction), while those that genuinely fold the spectrum are sinusoidal (sine-LoRA, loran), and even there the extra rank is overwhelmingly low-energy. For these we introduce Rank-Content Ablation (RCA), a within-checkpoint causal test that ablates the manufactured spectrum of a trained update while separating rank content from energy. Across 40 sine-LoRA checkpoints the manufactured rank is inert by a pre-registered load-bearing criterion: truncating back to the nominal rank is lossless ($\eta=1.00$), and the manufactured tail alone sits at the no-adapter floor. The verdict replicates on loran and across the full folding spectrum. From the training side, the first matched-per-arm-tuning evaluation finds no surviving advantage (a few tenths of a point at most, reproduced by zero-rank-lift controls, accuracy flat-to-decreasing in the rank-controlling frequency), on GLUE (DeBERTaV3) and commonsense (Llama-3.2-1B QLoRA) alike, where no adapter beats the no-adapter base. A rank-demanding probe confirms the test can fire (a genuine rank-64 adapter is catastrophic to truncate, $\eta=0$), while sine cannot use rank at all: it trains only where it manufactures no rank, and folds only where it cannot train. Our claim concerns nonlinear low-rank adapters for task adaptation; we neither test nor contest the signal-fitting regimes (e.g. implicit neural representations) where the construction originates and high-frequency structure may make rank genuinely load-bearing. Manufacturing rank with a nonlinearity inflates the spectrum, but on these tasks that rank is a spectator, out of the optimizer’s reach.