From Static Context to Calibrated Interactive RL: Mitigating Distribution Shift in Multi-turn Dialogue with Aligned Simulator

• URL: http://arxiv.org/abs/2605.26403v1
• Date: Tue, 26 May 2026 00:25:12 GMT
• Title: From Static Context to Calibrated Interactive RL: Mitigating Distribution Shift in Multi-turn Dialogue with Aligned Simulator
• Abstract summary: A long-standing goal of the research community is to develop highly interactive LLM-based dialogue agents.<n>Recent research focuses on optimizing policies based on fixed offline logs (Static Context RL) or using a prompt-based simulator (Interactive RL)<n>We show that both paradigms are fundamentally limited by context distribution shift.<n>We propose Calibrated Interactive RL, a unified framework that couples interactive RL with simulator alignment.
• Score: 33.62561779855603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A long-standing goal of the research community is to develop highly interactive LLM-based dialogue agents. Recent research focuses on optimizing policies based on fixed offline logs (Static Context RL) or using a prompt-based simulator (Interactive RL). In this work, we theoretically show that both paradigms are fundamentally limited by context distribution shift--a mismatch between dialogue histories observed during training and those encountered in real conversations. This shift compounds quadratically over turns and severely degrades dialogue quality. Specifically, we attribute this shift to two distinct sources: (i) policy-induced shift, arising from training on static histories rather than self-generated trajectories; and (ii) simulator-induced shift, stemming from discrepancies between simulated and real human behaviors. To address these challenges, we propose Calibrated Interactive RL, a unified framework that couples interactive RL with simulator alignment. By aligning the simulator with human interaction patterns, our approach reduces the sim-to-real gap and mitigates compounding distribution shifts. Experiments across multiple dialogue tasks confirm our theoretical analysis: (i) Interactive RL significantly outperforms the Static Context baseline by mitigating policy distribution shift; and (ii) calibrating simulators with our alignment method further bridges the sim-to-real gap, yielding state-of-the-art downstream performance.

Related papers

• Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions [62.99524765341952]
  Large language models are increasingly used to simulate social media users.<n>We study counterfactual context revision as a framework for auditing LLM-based stance simulation.<n>Results reveal effective and robust stance transitions in both text-only and multimodal strategies.
  arXiv  Detail & Related papers  (2026-06-04T17:41:54Z)
• Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors [61.610957638373826]
  We introduce a method to measure the distributional gap between real and simulated user behaviors.<n>Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation.<n>We show that most simulators behave similarly, while a few stand apart.
  arXiv  Detail & Related papers  (2026-05-08T15:09:25Z)
• MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignment [15.586402133245313]
  MUSE is a Chinese user simulation framework designed to generate human-like, controllable, and behaviorally consistent responses.<n>We propose Iterative Profile Self-Evolution and Role-Reversal Supervised Fine-Tuning to improve local response realism and human-like expression.<n>Experiments show that MUSE consistently outperforms strong baselines in both utterance-level and session-level evaluations.
  arXiv  Detail & Related papers  (2026-04-15T13:01:00Z)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces [81.41397370235102]
  We introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data.<n>We show that current models struggle to accurately simulate complex behaviors, with performance plateauing even as context windows expand.<n>This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
  arXiv  Detail & Related papers  (2026-04-09T15:26:21Z)
• Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation [0.7031557790463293]
  We introduce ContextSim, an agent framework that simulates believable user proxies by anchoring interactions in daily life activities.<n>To align preferences with genuine humans, we model agents' internal thoughts and enforce consistency at both the action and trajectory levels.
  arXiv  Detail & Related papers  (2026-01-26T05:01:00Z)
• See, Think, Act: Online Shopper Behavior Simulation with VLM Agents [58.92444959954643]
  This paper investigates the integration of visual information, specifically webpage screenshots, into behavior simulation via VLMs.<n>We employ SFT for joint action prediction and rationale generation, conditioning on the full interaction context.<n>To further enhance reasoning capabilities, we integrate RL with a hierarchical reward structure, scaled by a difficulty-aware factor.
  arXiv  Detail & Related papers  (2025-10-22T05:07:14Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• LitSim: A Conflict-aware Policy for Long-term Interactive Traffic Simulation [3.550570240658488]
  We propose a long-term interactive simulation approach that maximizes realism by minimizing the interventions in the log. Specifically, our approach primarily uses log replay to ensure realism and intervenes only when necessary to prevent potential conflicts. We then encourage interactions among the agents and resolve the conflicts, thereby reducing the risk of unrealistic behaviors.
  arXiv  Detail & Related papers  (2024-03-07T07:58:58Z)
• A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning [3.1515473193934778]
  Off-dynamics Reinforcement Learning seeks to transfer a policy from a source environment to a target environment characterized by distinct yet similar dynamics. We propose an innovative approach inspired by recent advancements in Imitation Learning and conservative RL algorithms.
  arXiv  Detail & Related papers  (2023-12-24T13:09:08Z)
• Replicating Complex Dialogue Policy of Humans via Offline Imitation Learning with Supervised Regularization [7.151589223349882]
  Policy learning (PL) is a module of a task-oriented dialogue system that trains an agent to make actions in each dialogue turn. Both supervised learning (SL) and reinforcement learning (RL) frameworks cannot imitate humans well. This study proposed an offline imitation learning model that learns policy from real dialogue datasets.
  arXiv  Detail & Related papers  (2023-05-06T09:27:58Z)
• VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction [50.986371459817256]
  We propose a novel textitVirtual InteRacTion mechanism, termed as VIRT, to enable full and deep interaction modeling in representation-based models. VIRT asks representation-based encoders to conduct virtual interactions to mimic the behaviors as interaction-based models do.
  arXiv  Detail & Related papers  (2021-12-08T09:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.