Sleep framework: A new way for LLMs to retain knowledge in continual learning

A closer look at Sleep framework: A new way for LLMs to retain knowledge in continual learning

What the Sleep framework is

This is a continual‑learning paradigm for large language models that adds a periodic "sleep" phase alongside the usual active or "wake" phase. During sleep the model pauses new inputs and focuses on offline memory consolidation and self‑improvement using techniques within the existing model architecture and training rules.

The framework uses a Mixture‑of‑Experts (MoE) memory design, temporary short‑term memory during active use, and automated procedures that transfer fragile in‑context knowledge into longer‑term parameters.

How the sleep and dreaming stages actually work

The sleep cycle has two main technical stages: consolidation and dreaming. Consolidation expands capacity (via MoE and low‑rank experts), then uses generalized knowledge distillation and reinforcement‑learning‑style imitation to seed long‑term parameters with selected short‑term information.

Dreaming generates synthetic training samples by randomly activating nonstandard experts, filters those samples with a gradient‑based selection step, and then refines the best samples via supervised fine‑tuning and RL evaluation to improve downstream performance.

Who this will be useful for

Researchers and engineers working on continual learning, memory consolidation, or model architectures that use Mixture‑of‑Experts will find the framework directly relevant.

It is also useful for anyone trying to reduce catastrophic forgetting or extend the effective retention of long‑context knowledge in deployed LLMs, since the framework is explicitly designed to retain facts and improve reasoning without changing core model code.

Questions people ask about Sleep framework: A new way for LLMs to retain knowledge in continual learning

Does the sleep framework change the model architecture?

The framework operates within a fixed model architecture but unlocks pre‑allocated MoE memory blocks and integrates new low‑rank experts during the sleep phase.

How does it avoid catastrophic forgetting?

It expands capacity during sleep to add new expert parameters, uses knowledge seeding (distillation and imitation) to transfer selected short‑term traces into long‑term modules, then prunes or resets temporary high‑frequency parameters.

What benefits does dreaming provide?

Dreaming mixes domains via random expert activation, generates and filters synthetic samples, and uses RL‑style refinement to discover latent patterns and improve downstream task performance.

Sleep framework: A new way for LLMs to retain knowledge in continual learning was shared on NexURLs by Nexis on October 31, 2026, in Science.