source: arxiv statistics ml: cora: per-slice coherent orthogonal rotation for svd-based low-rank adaptation
level: research
parameter-efficient fine-tuning often uses low-rank updates to pretrained weights. recent methods use the singular value decomposition of the base weight for initialization or subspace selection. these methods do not explicitly keep the coupled geometry between the pretrained left and right singular bases. recent minimum-perturbation theory shows that stable fine-tuning follows a coherent svd rotation. a single orthogonal matrix acts on both the left and right singular bases. this work proves a per-slice version of that idea.
each row slice of the weight matrix can be adapted by a shared orthogonal rotation on its left and right bases, plus a diagonal spectrum shift. the method is called cora, short for coherent orthogonal rotation adaptation. it applies per-slice orthogonal rotations and a per-layer diagonal scale to the rank-r update. this preserves the original singular vector relationships within each slice. the approach builds directly on the geometry of the pretrained model.
experiments show cora matches or exceeds the performance of existing low-rank adaptation methods while using fewer parameters. it works across vision and language tasks. the per-slice design adds flexibility without breaking the coherent structure. the method is simple to implement and adds little computational overhead. it provides a principled way to initialize and constrain fine-tuning updates.
why it matters: it offers a more parameter-efficient fine-tuning method that respects pretrained weight geometry, potentially reducing compute and memory costs for adapting large models.
source: arxiv statistics ml: cora: per-slice coherent orthogonal rotation for svd-based low-rank adaptation