level: research
continual fine-tuning of large language models with low-rank adapters often causes forgetting. when new tasks are added, standard lora-style methods stack updates on frozen weights, and later tasks can overwrite earlier ones. recolora addresses this by using a spectrum-aware recursive consolidation approach. it starts by initializing adapters from a randomized singular value decomposition of the pretrained weight. per-layer effective ranks are chosen using an elbow criterion, and the principal subspace is adapted before opening residual capacity.
before each new task, recolora re-decomposes the current effective weight instead of the original pretrained weight. this splits the model into a frozen residual, a slowly updated principal component, and a fresh adapter. the recursive consolidation means every task begins from a model that has already absorbed its predecessors. the principal component captures shared knowledge across tasks, while the fresh adapter learns task-specific details without disturbing what was already learned.
experiments on a six-task continual glue sequence across four 7-8b parameter backbones show recolora matches or exceeds the performance of training separate models for each task. it maintains accuracy on earlier tasks while learning new ones, effectively reducing catastrophic forgetting. the method works with different model architectures and does not require storing old data or expanding model size over time. its spectrum-aware rank selection adapts to each layer's needs, making efficient use of parameters.
why it matters: this approach lets ai systems learn new tasks sequentially without forgetting old ones, which is essential for building adaptable, long-lived models in production.