source: arxiv artificial intelligence: in-context reinforcement learning under non-stationarity: a survey
level: research
in-context reinforcement learning (icrl) lets pretrained models adapt to new tasks by using interaction history within their context window, without updating weights. most surveys focus on stationary settings where the task stays the same. this paper surveys icrl under non-stationarity, where the reward or transition dynamics change over time. the authors organize existing work by how context is built and used, covering decision-pretrained transformers, algorithm distillation, long-context meta-rl, and retrieval-augmented agents.
the survey categorizes approaches into four groups: sliding-window methods that forget old context, memory-augmented methods that store and retrieve relevant past experiences, change-detection methods that explicitly identify when the task shifts, and meta-learning methods that learn to adapt quickly from limited new data. each group has different strengths and weaknesses in handling abrupt or gradual changes. the paper also discusses evaluation benchmarks and metrics used to measure performance in non-stationary settings.
key challenges include balancing stability and plasticity, managing context length, and detecting task boundaries without explicit signals. the authors highlight open problems such as scaling to high-dimensional state spaces, combining multiple adaptation strategies, and theoretical understanding of when icrl fails under distribution shift. they suggest future work on more realistic non-stationary benchmarks and integrating icrl with continual learning techniques.
why it matters: understanding how to make ai agents adapt to changing environments without retraining is crucial for real-world applications like robotics, finance, and autonomous driving.
source: arxiv artificial intelligence: in-context reinforcement learning under non-stationarity: a survey