Manufactured Rank Is Real but Unrealized: An Optimization Barrier in Nonlinear Low-Rank Adapters for Task Adaptation

Abstract

Low-rank adapters can be made more expressive by inserting an element-wise nonlinearity between the low-rank factors: methods such as sine-LoRA and loran compute $\Delta W = \gamma\,\phi(\omega BA)$, which provably raises the realized update’s rank beyond the nominal ceiling at no extra parameter cost. We study what it takes to realize and exploit this manufactured rank in task adaptation, and give three results. First, an optimization-barrier argument: producing genuine extra rank requires driving the fold argument $\omega BA$ into the nonlinear range, but the loss curvature there grows as $\Theta(\omega^2)$, and we find stable first-order training in practice stays in the near-linear regime in which $\sin(\omega BA)\approx\omega BA$, which fails to take advantage of the manufactured rank. Second, a within-checkpoint ablation, Rank-Content Ablation (RCA), finding that the rank actually realized in training contributes no measurable task signal at convergence under standard settings: under the default recipe, truncating a trained update back to its nominal rank is lossless, and the same holds for a second method. Third, a synthesis of the conditions a method must satisfy to put manufactured rank to work; under matched tuning we find no detectable advantage over a well-tuned linear adapter. Run from scratch in the signal-fitting regime the construction comes from (3D occupancy, with each construction’s faithful recipe), the same ablation finds the manufactured rank is load-bearing, so the inert versus load-bearing boundary tracks whether training reaches the folding regime and whether the task rewards rank, rather than being a property of the construction itself. We release code and pre-registered analysis, and scope our adaptation claims to the settings we measure.