Manufactured Rank Is Real but Unrealised

This post is the companion to a preprint, Manufactured Rank Is Real but Unrealized: An Optimization Barrier in Nonlinear Low-Rank Adapters for Task Adaptation. Code, pre-registration, and the data artefact are on GitHub at captainpete/rank-content-ablation.

LoRA fine-tunes a frozen weight $W_0$ by adding a low-rank update $\Delta W = \tfrac{\alpha}{r}BA$, where $B$ and $A$ are thin matrices of rank $r$.¹ The rank is both the constraint and the point of the method: the update can move the weights in only $r$ directions, and that is what keeps the parameter cost small. A recent line of work argues that the ceiling can be lifted for free. Put an element-wise nonlinearity between the factors, $\Delta W = \gamma\,\phi(\omega BA)$, and the resulting matrix has rank well above $r$ with no parameters added. Sine-LoRA, which takes $\phi=\sin$, is the most developed version, and it reports consistent gains at the same parameter budget.²

I came to this hoping it would hold. The claim, if it is true, is not really about LoRA: it says a cheap nonlinearity can manufacture capacity that a linear map of the same size cannot reach, and that is the sort of thing you would want to use wherever a low-rank bottleneck shows up. So I set out to reproduce the effect, expecting to confirm it and then look for where else it applied, though what began as reproducing it soon turned into testing whether the extra rank was doing any work at all.

The case for the trick rests on two things, a theorem and a correlation. The theorem is solid: applying a sinusoid element-wise to a low-rank matrix provably raises its rank.³ The correlation is real as well, in that across configurations the updates with higher realised rank tend to score better. What gets inferred from the pair is that the manufactured rank is what produces the gain, and that inference is the part nobody had tested directly.

What manufacturing rank requires

It helps to start with the mechanism. Applying a sinusoid to a single number does nothing to rank; what matters is how $\sin$ acts on the entries of the matrix $BA$. Near zero, $\sin(\omega x)\approx\omega x$, so the sinusoid reproduces its input up to a scale factor, leaving the directions and the rank unchanged. New rank appears only once the argument $\omega x$ is large enough that the curve bends, past $\pi/2$, where $\sin$ stops being monotone and begins folding its input back on itself. A fold can send two different entries of $BA$ to the same output, or two equal entries to different outputs, and that re-routing is what fills in new singular directions.

So the rank is not quite free. It has to be paid for in amplitude: the entries of $\omega BA$ have to be large enough to reach the bend, and how far they are driven, $|\omega BA|$, is what I will call the fold depth. The slider below shows the trade-off. At low frequency the realised map (green) sits on the linear reference (grey) across the whole range the trained factors occupy, and no rank is added; raising the frequency makes the curve fold, but notice what that takes.

fold depth reached |ωBA| 0.35

no rank manufactured sine acts as a rescaled linear map; peak deviation from linear inside the band is 2%

Keep the shaded band in mind, because the argument keeps returning to it. It marks where the entries of a trained $BA$ actually sit. On a real adapter some of those entries do reach past the bend, so the fold genuinely happens and new directions appear. What the rest of the post turns on is how much of the update those directions actually carry.

A tail of tiny singular values

Suppose the fold does happen, and look at what it produces. There are two ways to count the rank of a matrix. The numerical rank counts singular values above a small threshold, so any direction with a non-zero foothold counts, however faint. The stable rank weighs each direction by its energy, $\rho = \lVert\Delta W\rVert_F^2 / \sigma_1^2$, so a direction carrying almost no Frobenius mass barely registers.⁴ On a trained sine adapter the two measures come apart.

frequency ω 101001000

127 numerical rank
count the bars

1.31 stable rank
weigh the bars

2% of the energy
beyond rank 4

The count climbs and the energy does not. At a frequency that trains well, the numerical rank runs into the hundreds while the stable rank stays barely above one, because the manufactured directions form a long tail of singular values, each one tiny beside the top handful. About two percent of the update’s energy sits beyond the nominal rank $r$.⁵ A correlation between rank and accuracy cannot say whether that two percent is a small real signal or just low-energy ripple, which is the kind of question only an intervention can answer. And there is plenty for the intervention to rule out, because the sinusoid changes more than the rank: near the origin it rescales the effective learning rate, the $\lvert\sin\rvert\le 1$ bound caps the update, and the $\cos(\omega BA)$ factor gates the gradient by a data-dependent amount.⁶ Any of these could produce an optimisation-side benefit while the extra rank comes along for the ride.

It helps to narrow the target first, because the “nonlinearity as rank” banner is broader than the methods it actually fits. Some adapters filed under it never raise the realised rank at all: the generative RBF construction genLoRA, named for the principle, proves $\mathrm{rank}(\Delta W)\le r$ as its own Proposition 3.1, because its nonlinearity acts on how the factors are generated and never on their product.⁷ The rank-mechanism question does not apply to it. The methods that genuinely fold the product are the sinusoidal ones, sine-LoRA and loran, and those are the two I test here.

Asking the update directly

The most direct way to find out whether a part of a trained model matters is to remove it and see what breaks. The update is a fixed matrix once training finishes, and its singular value decomposition gives its directions sorted by energy, so I can take a trained adapter, operate on that spectrum, bake the modified update back into the frozen weights, restore the trained head unchanged, and re-evaluate on held-out data. Nothing about training changes; only the realised $\Delta W$ moves. The battery, which I call Rank-Content Ablation, is four interventions, each aimed at a different question.

1. Truncate to the nominal rank. Throw away every direction beyond the original $r$, keep the top-$r$ part, and re-score. This asks the headline question head-on: if the manufactured rank is carrying the load, deleting it should hurt. On its own it is not conclusive, because the discarded tail holds energy as well as rank, so a drop could mean lost magnitude rather than lost content.

2. Remove the same energy from random directions. This is the intervention that separates rank from energy. Rather than deleting the manufactured directions, take an equal Frobenius mass out of random ones, leaving the top-$r$ subspace exactly intact, and set the two side by side.⁸ If the truncation hurts and this does not, the damage was the rank content specifically; if both hurt the same, it was only ever the energy. A rank-versus-accuracy correlation cannot draw that line; this intervention can.

3. Keep only the tail. Run the truncation in reverse: discard the top-$r$ part and evaluate the manufactured rank by itself. If it encodes anything the task uses, it should clear the no-adapter baseline; if it is a byproduct, it sits at the floor.

4. Graft the tail onto a plain LoRA adapter. Take the sine adapter’s tail spectrum, attach it to a genuinely linear adapter that never folded anything, and re-score. If an inflated spectrum helped in itself, adding one should help even here. This last one is a placebo, so it can only support the null, and it is kept out of the affirmative case.

Each intervention reads out as a single number, the retained gain fraction $\eta = \big(m(\widetilde{\Delta W}) - m_0\big) / \big(m(\Delta W) - m_0\big)$, where $m_0$ is the frozen model’s score with no adapter. An $\eta$ of one means the intervention changed nothing; an $\eta$ of zero means the score fell all the way back to the base. I fixed the decision rule in advance, assigning the load-bearing and byproduct outcomes to non-overlapping bands of $\eta$, so the result could not be reinterpreted after the fact.⁹

The result

Here is one trained sine adapter on CoLA, a representative case, with the interventions applied to it. The frozen model scores 18.2 and the intact adapter lifts that to 66.1; the question is how much of the lift each intervention leaves behind.

Truncating away the manufactured rank, and removing the same energy from random directions, both leave the score at the intact ceiling, so the manufactured rank carried none of the lift; the tail on its own sits back at the no-adapter floor.

Removing the manufactured rank costs nothing. Truncating back to rank $r$ reproduces the intact score to three figures, and stays there after the norm is rescaled, so the result is not a hidden magnitude effect. Removing the same amount of energy from random directions costs nothing either, which settles the confound: there is no energy penalty to account for, since energy taken out at random is as harmless as energy taken out along the manufactured directions. The manufactured tail on its own does not clear the no-adapter floor; evaluated alone, the rank the fold produced scores exactly what the untrained model scores. And grafting that tail spectrum onto a plain LoRA checkpoint, the fourth intervention, barely moves its score, a thousandth of a point on average, so the inflated spectrum is inert even on a genuinely linear adapter.

This is not a single lucky checkpoint. Across all forty trained sine adapters on the primary tier, truncation $\eta$ never drops below 0.995 and tail-only $\eta$ never rises above 0.001. Extending the test across the full frequency sweep leaves the reading unchanged, with truncation lossless at every frequency in it. More manufactured rank does not make any of it load-bearing. The result is not specific to the sine function either, since loran manufactures rank through a different element-wise map and comes back just as inert, truncating losslessly with its tail at the floor.¹⁰ On the FP16 GLUE tier the verdict is unanimous across every trained checkpoint I produced.

No gain to explain

All of the above is about mechanism, and it stands no matter how large the headline effect is, because each intervention is paired within a single checkpoint. But it is fair to ask whether the published advantage that motivated the exercise even survives a careful comparison. So I ran what is, as far as I can tell, the first matched-per-arm-tuning evaluation of sine-LoRA: nine arms at identical parameter budget, each tuned on its own grid, with the headline frequency frozen before any test-split scoring.

It does not: at its frozen frequency sine does not beat plain LoRA on any of the four tasks; plain LoRA at the next rank up matches or beats it everywhere, and the pre-registered inference finds no contrast significant.¹¹

The comparison that says more is the one against controls. Besides the rank, the sinusoid acts through the three optimisation-side channels from earlier, so I trained three linear arms, one reproducing each, none of them lifting the realised rank above nominal, and all three land on sine. The telling one is the gradient gate: its forward pass is exactly linear, $\gamma\,\omega BA$, while its backward pass is multiplied by $\cos(\omega BA)$, so it trains under sine’s gradient gating while folding nothing. That closes the most stubborn objection, that the manufactured rank was a useful scaffold during training even if idle at convergence: had it been, this arm, carrying sine’s training-time gradient but never folding, would fall short of sine, and it does not. Whatever small, task-dependent effect the reparameterisation has sits in these optimisation-side channels, not in the rank it manufactures.

A final view of the matched-tuning runs sweeps the frequency that sets how much rank is manufactured.

RTECoLAMRPCSTS-B

As the frequency rises, the trained stable rank climbs, but accuracy does not climb with it. On RTE and CoLA it falls; on MRPC and STS-B it peaks early and then drifts down. On every task the best setting is one of the lower-rank ones, and raising the frequency past it never helps. This mirrors the intervention result from the training side: the manufactured rank is inert within each checkpoint, and manufacturing more of it across checkpoints buys nothing.

The cross-architecture run adds one more wrinkle. On a 4-bit quantised Llama doing commonsense reasoning both findings replicate, and the no-adapter base puts them in context: on this subset fine-tuning mostly degrades the frozen model, and the only arm that holds the base is MoRA, the genuine linear high-rank construction, with sine among the worst. Where rank helps at all, an explicit linear method captures it, and sine’s nonlinearly-manufactured rank does not, for the same reason as everywhere else: it never reached the fold. In every setting I measured, no form of rank beat a well-tuned baseline, which is what you would expect once the adaptation tasks turn out to hold no high-rank structure for any method to use.

At this point I had two clean nulls, the per-checkpoint interventions and the matched-tuning comparison, and the obvious thing to worry about had shifted from sine-LoRA to the instrument itself.

Whether the test can fire

There is an obvious objection. If the manufactured tail holds only two percent of the energy, truncating it is cheap as linear algebra, and the lossless result might be fixed by the operating point rather than by any absence of content. A test that can only ever come back lossless would tell us nothing. So before treating the null as a fact about sine-LoRA, I needed to show the test can register a positive when rank genuinely matters.

Part of the answer is internal to these experiments. The lossless truncation is not simply an artifact of the tail being small: when the sinusoid is driven hard enough that the manufactured spectrum carries real energy, to a stable rank around seventeen at the highest frequency, truncating it is still lossless, so the operating point is not what guarantees the result. And $\eta$ is not pinned at one in general, since the same truncation applied to MoRA, a genuinely high-rank linear adapter, does lose ground, because MoRA puts real energy into its extra rank.¹² But a decisive check needs a task where rank is clearly the binding constraint, and adaptation benchmarks are not. They are low-rank problems by nature, which is why plain LoRA works on them in the first place, so there is no load-bearing rank for any method to lose. I needed a task that demands rank by construction.

Associative memorisation is such a task. Asked to store $N$ random key-to-value pairs in a low-rank update, a frozen model can hold a number of pairs that scales with rank: a rank-4 adapter saturates near a thousand, a rank-64 adapter holds several thousand. Train a rank-64 adapter to memorise 4096 pairs, confirm it recalls them perfectly, then truncate it back to rank 4, and recall collapses from 100 percent to zero, $\eta=0$. So the test does fire: it returns one on GLUE because GLUE has no high rank to lose, and zero as soon as the content beyond rank $r$ is genuinely carrying the task.

The optimisation barrier

If sine’s nonlinearity really does buy capacity beyond the nominal rank, a task that rewards that capacity should be where it pays off. So I held the memorisation demand fixed and swept the frequency, watching whether the adapter could still be trained, how much realised rank it carried, and how much rank it advertised.

The two curves move apart rather than together. Where sine trains, recall holds near 100 percent, but the fold depth is tiny and the realised stable rank stays at the level of a plain rank-4 adapter, even while the advertised numerical rank climbs steadily. Raise the frequency far enough to manufacture rank that carries real energy, and training breaks down: recall crumbles through a band where individual seeds split, and by the time the fold is genuinely deep the training loss has risen to the no-learning ceiling, the value of predicting nothing.¹³ Realised rank shows up only at the far extreme, where the adapter cannot be trained at all. The trainable regime and the folding regime do not overlap at any frequency.

That gap has a cause worth naming, because it is what shapes the whole result. Folding the spectrum means driving the sinusoid past its bend, and the curvature that does the bending, $\sin''=-\omega^2\sin$, scales the curvature of the loss by $\omega^2$: the deeper the fold, the more ill-conditioned the landscape, until a step size that trains a linear adapter is far too large to descend it.¹⁴ Producing rank that carries energy needs exactly the frequencies at which gradient descent can no longer follow the sinusoid, and following the sinusoid needs exactly the frequencies at which it folds too gently to make any rank. This is the optimisation barrier, and it is not a flaw peculiar to sine: the same periodicity that makes these activations so good at representing high-frequency signals is what makes them hard to train through when you put them between the factors.

The barrier is a fact about how the adapter is trained, not a law about the construction, and that leaves a way around it. If training could start already inside the folding regime, from a well-conditioned scale rather than climbing into it from zero, the curvature would be there from the first step and gradient descent could work with it. A network trained from scratch can be initialised that way. An adapter cannot: the convention that makes it an adapter, starting from $\Delta W\approx 0$ so the frozen model is unchanged at initialisation, pins it at $\omega BA=0$, the one place the fold has not begun. So on a task that genuinely rewards going past rank 4, sine under the standard recipe never reaches the rank that would help, while a plain linear rank-64 adapter reaches it easily. That does not close off the idea; it points to exactly where the idea should work, which is the setting it came from.

Where the rank is real

Everything so far is set in adaptation, where the fold never gets deep and the task never asks for rank. Both of those lift in the setting sine activations actually come from, which is fitting a signal directly. There the network is trained from scratch, so it can be initialised inside the folding regime, and the target is itself high-frequency, so deep folds are exactly what the task rewards. Sine-LoRA’s own paper shows the construction working there, fitting the occupancy field of a 3D shape with a coordinate network.¹⁵ I took the same Rank-Content Ablation probe, unchanged, and pointed it at that setting.

The probe inverts. Fitting a detailed shape at rank one, a plain low-rank layer reaches an intersection-over-union of about 0.78, while the sine layer, initialised into the fold, reaches 0.99, close to the full-rank ceiling and matching the gain the original authors report. Now truncate that trained sine layer back to its nominal rank, the operation that was lossless on every adapter, and the fit collapses: IoU falls to 0.15, a retained gain of essentially zero.¹⁶ The matched-energy control removes less than the truncation does, so the loss is specific to the manufactured content rather than its magnitude, the same contrast that read the other way in adaptation. And the plain low-rank layer, which never folded anything, is untouched by the cut in both settings.

The same truncation probe, in both regimes. In adaptation it is lossless, so the manufactured rank carried none of the lift; in signal fitting it collapses the fit, so the manufactured rank carried it.

So the manufactured rank is real, and it is load-bearing, once two conditions both hold: training has to reach the folding regime, and the task has to reward the rank that folding produces. Signal fitting meets both. Adaptation meets neither, and the two shortfalls are independent, since even an adapter carried into the fold would find a low-rank task with no use for the rank waiting there, and even a task that rewarded rank would leave the standard adapter unable to reach the fold to make any. The same test reads one where the rank sits idle and zero where it carries the fit, which is exactly what lets it tell the two apart. The boundary between inert and load-bearing runs along the task and the training, not along the construction.

What this does not claim

It is easy to over-read a clean result, so I should be clear about the edges. The claim is about nonlinear low-rank adapters used for task adaptation, under standard first-order training, and each of those qualifiers earns its place. The barrier is stated for AdamW at a fixed learning rate, the way the method is actually deployed, and it is not a proof that no optimiser could cross it; a curvature-aware step, an annealed frequency, or an initialisation that starts an adapter already inside the fold could each in principle make the rank reachable, and setting out those conditions is part of what the paper tries to do.¹⁷ The matched-tuning comparison leans on a reconstructed recipe, since the original GLUE setup is not fully reproducible, so it is the within-checkpoint interventions I would stand on hardest.¹⁸ And the signal-fitting result is one regime, occupancy fields on coordinate networks, rather than the whole fitting literature; it shows the rank can be load-bearing, not that it is wherever the construction is used.

The most reusable part is the test rather than the verdict on any one method. Rank-Content Ablation needs only a trained update whose realised rank exceeds its nominal rank, and the matched-energy control is what turns an ambiguous “truncation hurts or helps” into a clean separation of rank content from energy. If a method claims the rank-$r$ ceiling is the bottleneck, the rank it adds ought to survive having its content removed with its energy held fixed, and it ought to earn that rank on at least one task whose adaptation actually demands it.

Why I went looking

I was excited by the original paper, more than a first read of this one might suggest. The idea seemed to point past LoRA at something general about nonlinearity and rank, and I picked it up meaning to confirm it and then chase where it led, and where it led was not where I expected. Chasing the effect turned into building a test for it, and the test kept saying the same thing about adaptation: the manufactured rank is genuinely there in the spectrum and does no work, because the task asks for only a handful of directions and the optimiser never drives the fold deep enough to make more.

For a while I read that as the idea not panning out. What changed my reading was pointing the same test at the setting the construction came from and watching it invert: the rank is real, and where training reaches the fold and the task rewards it, it does what the original work claimed. That is a sharper and more useful thing to know than a flat yes or no.

I still think it was the right thing to chase. Most of what moves this field forward looks like sleight of hand until it works, the chain rule making deep networks trainable at all being the standard example, and a construction with that same smell, a simple move that seems to buy something for nothing, is one I will always want to run down, because once in a while one of them is real. This one is real; it is just unrealised where I first went looking for it. The magic I went looking for was sitting one regime over the whole time, in the signal-fitting problem the sinusoid was built for.

Footnotes

1. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” in International Conference on Learning Representations (ICLR), 2022. The ratio $\alpha/r$ is a fixed scale; what training learns is the pair of factors $B$ and $A$. ↩

2. Y. Ji, H. Saratchandran, C. Gordon, Z. Zhang, and S. Lucey, “Efficient Learning with Sine-Activated Low-Rank Matrices,” in International Conference on Learning Representations (ICLR), 2025, arXiv:2403.19243. The construction is $\Delta W = \gamma\sin(\omega BA)$, with $\gamma$ a fixed amplitude and $\omega$ a frequency that sets how hard the input is driven into the sinusoid’s nonlinear region. ↩

3. The statement is that applying $\sin$ element-wise to a rank-$r$ matrix produces a matrix whose rank can exceed $r$, by expanding each entry through the sinusoid’s Taylor series and noting the higher-order terms introduce new outer products. The theorem is about possibility, not about how much energy those new directions carry, and that distinction is the gap the rest of the post measures. ↩

4. Numerical rank is the count of singular values above an absolute tolerance, here $10^{-5}$, and is sensitive to where that threshold sits. Stable rank, $\rho = \lVert\Delta W\rVert_F^2/\sigma_1^2 = (\sum_i \sigma_i^2)/\sigma_1^2$, is threshold-free and equal to one for any rank-one matrix, rising only as energy spreads across directions. The two answer different questions: how many directions are non-empty, and how many directions actually hold the weight. ↩

5. On DeBERTaV3-base at the published recipe, the trained fold depth $|\omega BA|$ reaches about $2.9\pi$, well past the $\pi/2$ where $\sin$ turns over, so the fold genuinely happens; the update carries roughly two percent of its Frobenius energy beyond rank $r$, and its stable rank reaches about $1.8$ times plain LoRA’s. There is real supra-nominal rank to interrogate; it just carries almost no energy. ↩

6. rsLoRA isolates the effective-learning-rate channel that $\omega$ also modulates [D. Kalajdzievski, “A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA,” 2023, arXiv:2312.03732], and OP-LoRA shows that over-parameterising the optimisation of a low-rank update, rather than its rank, can improve adaptation on its own [P. Teterwak, K. Saenko, B. A. Plummer, and S.-N. Lim, “OP-LoRA: The Blessing of Dimensionality,” 2024, arXiv:2412.10362], which is the direct precedent for treating the benefit as optimisation-side and for the gradient-gate control used later. ↩

7. Y. Ouyang, S. Li, H. Wang, X. Luo, Z. Hu, Y. Song, Q. Qin, Y. Li, and R. Li, “Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions,” 2026, arXiv:2602.05709. The generative RBF adapter forms $\Delta W = \sum_{i=1}^{r} f^B_i(Z_B)\,f^A_i(Z_A)^\top$, a sum of $r$ rank-one terms whose factors are synthesised nonlinearly from latent vectors. Because the nonlinearity acts on the generation of the factors and never on their product, the update is a sum of $r$ outer products and its rank is at most $r$ by construction. Whatever benefit it has, supra-nominal realised rank cannot be the lever, because it has none. ↩

8. Concretely: draw a Gaussian matrix $N$, project the top-$r$ row and column spaces out of it so it cannot disturb the kept subspace, rescale it to carry exactly the Frobenius norm of the discarded tail, and subtract it. The intact top-$r$ part survives untouched while an equal amount of energy is destroyed in content-free directions. Reported over several noise seeds. The contrast between this and plain truncation is the load-bearing test for rank content; if both are lossless, the truncation result cannot be an energy artifact. ↩

9. The bands: the load-bearing hypothesis required truncation to destroy at least half the gain, a positive content gap between random and directed removal, and an isolated tail above a small floor band; the byproduct hypothesis required near-lossless truncation, random and directed removal to agree, and the tail inside the floor band. Every observed value fell on the byproduct side, none close to the load-bearing threshold. The one pre-registered outcome the data could not resolve was a tight equivalence test, which the limited number of task clusters left underpowered; it is reported as unresolved rather than dropped from the record. ↩

10. G. Deng, M. Liu, D. Wu, Y. Li, and L. Song, “Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations,” 2025, arXiv:2509.21870. loran applies a structured sine map to the product, $\Delta W = \gamma(\beta\sin(\omega M)\odot M + M)$ with $M = BA$, reducing to plain LoRA as the amplitude $\beta\to 0$. At a premise-satisfying setting it folds to a trained stable rank comparable to sine’s and adapts strongly, so there is genuine manufactured rank to ablate. Pooled over twelve trained checkpoints the signature is identical to sine’s: truncation lossless, matched-energy removal lossless, tail at the floor. ↩

11. Inference is a per-task $z$-standardised model with cluster-robust standard errors clustered by task and Holm correction across contrasts, on twenty seeds; no contrast is significant, and a restricted wild cluster bootstrap-t agrees. With only four task clusters the design cannot reject at conventional levels by either route, so the honest reading is “no detectable advantage,” not “proven identical.” The descriptive paired differences trend slightly against sine, on the order of a few tenths of a point, in line with the broader finding that apparent LoRA-variant gains often dissolve under per-method tuning [Y.-A. Lee, C.-Y. Ko, P.-Y. Chen, and M.-Y. Yeh, “Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning,” 2026, arXiv:2602.04998]. ↩

12. T. Jiang, S. Huang, S. Luo, Z. Zhang, H. Huang, F. Wei, W. Deng, F. Sun, Q. Zhang, D. Wang, and F. Zhuang, “MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning,” 2024, arXiv:2405.12130. MoRA raises rank linearly at matched parameter count by reshaping the update into a square form, so it puts real Frobenius energy into its extra directions. Truncating it back toward rank one does cost gain, which is the within-experiment proof that the truncation is not trivially lossless. On GLUE that cost is modest because the tasks are themselves near rank-one; where rank genuinely binds, as in the rank-64 memorisation control, truncation is catastrophic. ↩

13. For a $V$-way next-token target, a model that has learned nothing sits at a cross-entropy of about $\ln V$ nats. Wherever sine trains, the loss is essentially zero; in the rank-manufacturing band the measured loss climbs back to that ceiling, so the failure is genuine non-optimisation rather than incomplete training. The optimiser makes no measurable progress at all, rather than slow progress. ↩

14. More precisely, in the folding range the gradient of the loss in the factors is Lipschitz with a constant of order $\omega^2$, so gradient descent is stable only for step sizes of order $\omega^{-2}$, the edge-of-stability bound inflated by the fold. A learning rate usable for adaptation sits far above that ceiling once $\omega$ is large enough to fold, so either $\omega$ is small and training stays near-linear, or $\omega$ is large and the high-curvature, sign-oscillating landscape is not descended. The $\omega^2$ scaling is a property of $\sin''$, so it survives an adaptive optimiser such as AdamW, which rescales the constant but not the dependence on $\omega$. This is the parameter-space analogue of the spectral bias of gradient descent, the same effect by which a plain network learns low frequencies first. ↩

15. The lineage runs through periodic-activation networks for implicit neural representations [V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein, “Implicit Neural Representations with Periodic Activation Functions,” in Advances in Neural Information Processing Systems (NeurIPS), 2020], where a sinusoidal activation is chosen because the target is high-frequency and the network is rewarded for representing fine detail. The from-scratch recipe uses a SIREN-style initialisation, which scales the factors so the sine argument starts at a controlled order-one magnitude, the well-conditioned scale from which training enters the fold stably rather than overshooting into the oscillating regime. ↩

16. The task is a coordinate network fit to the binary occupancy field of three shapes from the Stanford 3D Scanning Repository, reproducing sine-LoRA’s own occupancy experiment. Pooled over three shapes and five seeds, truncating the trained sine layer back to nominal rank returns a median retained gain of about zero, inside the pre-registered load-bearing band, against a plain low-rank control that truncation leaves untouched. The advantage is specific to low nominal rank on detailed shapes; at rank two and above a plain low-rank layer already saturates, so no construction gains, which mirrors the original trend. ↩

17. The paper states three, and the constructions studied meet none of them under standard training. The rank has to be reached without destabilising training, which the $\omega^2$ curvature blocks for a fixed-step optimiser and which a curvature-aware step, an annealed frequency, a landscape-lowering linear skip, or a pre-folded initialisation might clear; the realised rank has to carry task signal, which Rank-Content Ablation checks directly; and the evaluation has to sit on a task where rank is the binding constraint, since standard benchmarks are solved by near-rank-one updates. For a practitioner choosing an adapter today, the realised rank of a nonlinear low-rank adapter is, under standard training, that of a well-tuned linear one. ↩

18. The original sine-LoRA GLUE backbone is not identifiable from the paper, which names a model not in standard use and cites a sentence-embedding paper for it, while the reported scores fingerprint a DeBERTaV3-base recipe, which I adopt. The published recipe does not train on actual RoBERTa-base: at every stable learning rate the fold depth stays below the bend, so on that backbone the sine adapter is a linearly-rescaled LoRA with no manufactured rank to test, a scope condition that itself dictated running the interventions on DeBERTaV3. All deviations from the reconstructed recipe are logged in the repository. ↩

@misc{hollows2026manufact,
  author = {Hollows, Peter},
  title  = {{Manufactured Rank Is Real but Unrealised}},
  year   = {2026},
  month  = jul,
  url    = {https://dojo7.com/2026/07/11/manufactured-rank-is-real-but-unrealised/}
}