Do Modules Stay in Their Lane? Role Drift in Compound LLM Systems

• URL: http://arxiv.org/abs/2607.21627v1
• Date: Tue, 07 Jul 2026 22:31:23 GMT
• Title: Do Modules Stay in Their Lane? Role Drift in Compound LLM Systems
• Abstract summary: We identify Role Drift, a failure mode in which modules deviating from their assigned roles remain invisible to system-level evaluation.<n>We propose Role Anchor, a regularizer that modulates how much each module deviates from its assigned role during end-to-end training.
• Score: 0.66816887202191
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: End-to-end reinforcement learning can improve the accuracy of compound LLM systems, but it does not constrain how modules divide labor internally. We identify Role Drift, a failure mode in which modules preserve or improve end-task performance while deviating from their assigned roles through role-violating shortcuts that remain invisible to system-level evaluation. To make role drift observable and controllable, we propose Role Anchor, a regularizer that modulates how much each module deviates from its assigned role during end-to-end training. The key idea is to preserve how the role prompt shifts the module's next-token predictions relative to a neutral prompt, which serves as a proxy for the role's intended effect during training. Experiments on two compound LLM pipelines reveal role drift that accuracy alone fails to detect: a decomposer meant to split a question into sub-questions for a separate solver instead plants the answer in them, and a reader meant to answer from retrieved passages instead falls back on parametric memory. In fact, on the decomposer pipeline this shortcut drives most of the apparent RL gain: 86% of it vanishes once the decomposer is held to its role, indicating that terminal accuracy alone can badly overstate how much a compound system has genuinely learned. Across both pipelines, Role Anchor mitigates role drift at a tunable accuracy cost that varies by pipeline and anchor strength. Additional gradient analysis suggests that the regularizer reduces alignment with the role-drift direction rather than simply suppressing learning.

Related papers

• Revisiting Complete Reasoning Traces for Post-Training [61.76889440384448]
  Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability.<n>We show that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation.
  arXiv  Detail & Related papers  (2026-09-07T06:46:17Z)
• TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs [83.78708832901194]
  Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild.<n>However, real-world deployment remains challenging: modalities may be missing or noisy at test time.<n>We propose a solution in which a frozen teacher guides an adaptive low-rank student via self-distillation.<n>Experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation.
  arXiv  Detail & Related papers  (2026-08-18T23:40:28Z)
• A Graph-Based Reinforcement Learning Framework for Structured Drift Diagnosis and Recovery in Autonomous LLM Agents [0.5249805590164902]
  Existing approaches address drift at the prompt level but lack structured mechanisms for step-level detection, risk assessment, and recovery decision.<n>This work targets a plug-and-play recovery module instead.<n>It introduces a graph-based framework in which a single small language model is trained via reinforcement learning to specialize at each node of a recovery graph.
  arXiv  Detail & Related papers  (2026-08-14T09:12:14Z)
• Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning [2.749219725903528]
  Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially.<n>LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately.<n>Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting.<n>Latent-LoRA is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task.
  arXiv  Detail & Related papers  (2026-07-26T20:42:56Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations [41.8393565183611]
  We show that on-policy distillation (OPD) steers the student toward correct reasoning paths via dense token-level guidance, without expanding capability ceiling.<n>The Student-Teacher Mismatch occurs when a large teacher-student distributional gap causes the guiding signal to misalign with task correctness, steering exploration in counterproductive directions.<n>Length Exploitation arises when the aggregated token-level objective creates length-dependent shortcuts, allowing the student to game the reward landscape through response truncation or redundant padding.<n>To tame these pathologies, we investigate lightweight signal regulations: advantage clipping and log-scale compression,
  arXiv  Detail & Related papers  (2026-07-15T02:52:37Z)
• Purified OPSD: On-Policy Self-Distillation Without Losing How to Think [52.879387744920415]
  We find that on-policy self-distillation (OPSD) consistently fails on long chain-of-thought (long-CoT) reasoning models.<n>We propose a two-step solution: first, we construct a reference-only teacher to isolate the non-transferable component of the supervision signal.<n>Second, we use pointwise mutual information (PMI) as the mechanism to transform this residual into a well-formed PMI target distribution.
  arXiv  Detail & Related papers  (2026-07-02T14:33:07Z)
• Dynamic Rollout Editing for Reducing Overthinking in RL-Trained Reasoning Models [102.76983747945836]
  Long-form chain-of-thought reasoning can improve performance on complex tasks.<n>But models often continue generating unnecessary reasoning after a correct answer has emerged.<n>We study this phenomenon from the perspective of GRPO-style reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-16T13:10:30Z)
• When Roles Fail: Epistemic Constraints on Advocate Role Fidelity in LLM-Based Political Statement Analysis [0.0]
  This paper provides the first systematic empirical test of the assumption that models will reliably maintain their assigned roles.<n>We measure role fidelity across 60 political statements using four metrics: Role Drift Index (RDI), Drift Distance (EDD), Directional Drift Index (DDI), and Entropy-based Role Stability (ERS)<n>We identify two failure modes - the Epistemic Floor Effect (fact-check results create an absolute lower bound below which the legitimizing role cannot be maintained) and Role-Prior Conflict (training-time knowledge overrides role instructions for factually unambiguous statements)
  arXiv  Detail & Related papers  (2026-04-29T21:50:28Z)
• Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning [79.88942231770629]
  Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs)<n>We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR)<n>We synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement.
  arXiv  Detail & Related papers  (2026-04-08T00:53:29Z)
• Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behavior [58.751981587234916]
  This paper formalizes the Refinement Provenance Inference (RPI) audit task as Refinement Provenance Inference (RPI)<n>We propose RePro, a logit-based framework that fuses teacher-forced likelihood features with logit-ranking signals.<n>During training, RePro learns a transferable representation via shadow fine-tuning, and uses a lightweight linear head to infer provenance on unseen victims without training-data access.
  arXiv  Detail & Related papers  (2026-01-05T10:16:41Z)
• The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) [15.48684126686974]
  We show that fine-tuned large language models often rely on two proxies for role identification.<n>We propose reinforcing emphinvariant signals that mark role boundaries by adjusting token-wise cues in the model's input encoding.
  arXiv  Detail & Related papers  (2025-05-01T16:06:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.