level: research
reinforcement learning research often uses benchmark simulators to approximate sequential decision-making in deployment settings. however, the drive for high scores can shift focus toward merely solving the simulator itself. this paper contends that solving simulators and using them as proxies for learning in deployment are fundamentally different research questions. the authors stress that researchers need to recognize this distinction to avoid conflating the two goals.
the two use cases impose different constraints on how agents interact with the simulator. when solving a simulator, any method that boosts performance is fair game, even if it exploits simulator-specific quirks. in contrast, using a simulator as a proxy requires algorithms that would work in a real deployment, limiting access to privileged information or unrealistic actions. the paper discusses how these differences affect algorithm design and evaluation, and warns that ignoring them can lead to misleading progress claims.
the authors propose clearer experimental practices to separate these objectives. they suggest that papers should explicitly state whether they aim to solve a simulator or use it as a proxy, and that reviewers should hold work accountable to the stated goal. this clarity would help the field advance general-purpose decision-making rather than overfitting to benchmarks. the paper also touches on how current benchmarks often blur this line, making it hard to assess true algorithmic progress.
why it matters: distinguishing these goals prevents wasted effort on algorithms that only work in simulation and helps ensure rl research translates to real-world applications.