source: Google Research: The sleep framework: A new way for LLMs to retain knowledge in continual learning
level: research
google research proposes a sleep framework for large language models. it adds an offline sleep phase to the usual active wake phase. during sleep, the model consolidates short-term context into long-term parameters. this lets llms keep learning new facts and skills without catastrophic forgetting. the approach mimics how human brains stabilize memories during rest. it uses mixture-of-experts blocks and self-generated practice examples. the model updates weights only in pre-allocated memory blocks, so core code stays unchanged.
the method shows strong results on several benchmarks. for long-context understanding on babilong, it scales near-perfectly up to 10 million tokens, beating gpt-4 and titans. on factual knowledge tests like squad, it scores 48.9% single-task and 46.2% continued pre-training, outperforming supervised fine-tuning and seal. on math reasoning, it reaches 79.2% on aime 24, 69.0% on aime 25, and 46.1% on hmmt 25. for continual translation in unseen languages like manchu, it keeps near-zero performance drop while baselines forget severely.
the sleep framework has two stages: memory consolidation and dreaming. consolidation expands capacity with low-rank experts and distills short-term memory into long-term parameters. dreaming generates synthetic samples with random expert selection and filters them by gradient usefulness. reinforcement learning refines the model on top samples. the theory shows consolidation is a low-rank compression problem, and dreaming improves gradient geometry. this could lead to ai systems that learn continuously after deployment, reducing the need for costly retraining.
why it matters: this lets deployed llms update their knowledge continuously without forgetting old skills, which is key for real-world ai that must adapt to new information.
source: Google Research: The sleep framework: A new way for LLMs to retain knowledge in continual learning