Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training

• URL: http://arxiv.org/abs/2608.08224v1
• Date: Sat, 08 Aug 2026 16:44:47 GMT
• Title: Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training
• Abstract summary: Reinforcement learning post-training unlocks complex reasoning in LLMs.<n> benchmark scores reveal only whether a model improved, not what changed inside it, nor how it splits finite capability across tasks.<n>We challenge this activation-centered account by separating activation from control.
• Score: 5.633817116718103
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning post-training unlocks complex reasoning in LLMs. Yet benchmark scores reveal only whether a model improved, not what changed inside it, nor how it splits finite capability across tasks. A representative interpretability line attributes the success of RL fine-tuning to stronger and more diverse circuit activation. We challenge this activation-centered account by separating activation from control: an activated circuit need not control the post-training reward gain. Adapting Metabolic Control Analysis, we define the Post-training Control Coefficient to measure component control over reward gain and arrange these coefficients by task family into a control matrix, paired with an activation-magnitude matrix. We call cross-task control concentration the Shared Control Bottleneck and the difference between activation and control concentration the Activation-Control Gap. This reveals that highly shared activations can coexist with task-specific control, while a small gap indicates that control has collapsed onto a shared direction and lost task specificity. To reduce this collapse, we regularize the post-training loss with the Shared Control Bottleneck and propose Control-Diverse Reinforcement Fine-Tuning (CD-RFT). The exact regularizer gradient requires second-order automatic differentiation incompatible with flash attention, so we derive a first-order proxy with worst-case overhead below eight percent. On Qwen2.5-7B, CD-RFT achieves the largest control decoupling and improves multi-task capability over matched GRPO across mathematics, code, and logic. The no-KL variant leads on pass@1, and the KL-penalized variant leads on large-k pass@k coverage that KL otherwise degrades. Together, these results show that the Shared Control Bottleneck is both a mechanistic diagnostic and a training regularizer, and that control decoupling and capability gains transfer to Llama-3.2-3B.

Related papers

• Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning [0.05371337604556311]
  Large language model (LLM) agents are usually improved by changing prompts, calibrated models, or hand-written, while the execution harness around the model is treated as fixed infrastructure.<n>We formalize harness operation as a finite-horizon Harness MDP, where a lightweight controller selects structural execution actions while the LLM executor remains frozen.
  arXiv  Detail & Related papers  (2026-07-05T22:11:18Z)
• A Systematic Approach to Multi-Agent AI from Advanced Regulatory Control Theory: Safe and Auditable LLM Operator Agents for Process Control [0.2864713389096699]
  It is possible to hypothesise that a multi-agent reformulation under process-control principles offers a route to address those points.<n>The present work proposes such a reformulation, derived from Advanced Regulatory Control (ARC) theory.
  arXiv  Detail & Related papers  (2026-06-29T20:07:47Z)
• Iterative Learning Control-Informed Reinforcement Learning for Batch Process Control [17.081987402115477]
  This study introduces an Iterative Learning Control-Informed Reinforcement Learning (IL-CIRL) framework for training DRL controllers in dual-layer batch-to-batch and within-batch control architectures for batch processes.<n>The proposed method incorporates Kalman filter-based state estimation within the iterative learning structure to guide DRL agents toward control policies that satisfy operational constraints and ensure stability guarantees.
  arXiv  Detail & Related papers  (2026-03-16T12:17:43Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Verifying Closed-Loop Contractivity of Learning-Based Controllers via Partitioning [52.23804865017831]
  We address the problem of verifying closed-loop contraction in nonlinear control systems whose controller and contraction metric are both parameterized by neural networks.<n>We derive a tractable and scalable sufficient condition for closed-loop contractivity that reduces to checking that the dominant eigenvalue of a symmetric Metzler matrix is nonpositive.
  arXiv  Detail & Related papers  (2025-12-01T23:06:56Z)
• Attention on flow control: transformer-based reinforcement learning for lift regulation in highly disturbed flows [0.0]
  We learn an effective control strategy for regulating aerodynamic lift in arbitrarily long gust sequences via pitch control.<n>We show that the learned strategy outperforms the best proportional control, with the performance gap widening as the number of gusts increases.
  arXiv  Detail & Related papers  (2025-06-11T20:14:28Z)
• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution [51.83951489847344]
  In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
  arXiv  Detail & Related papers  (2024-04-05T17:58:37Z)
• C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory [20.38647732528661]
  Generative Adversarial Learning (GAIL) trains a generative policy to mimic a demonstrator. It uses on-policy Imitation Reinforcement Learning (RL) to optimize a reward signal derived from a GAN-like discriminator. Recent work has shown that control theory can help with the convergence of a GAN's training.
  arXiv  Detail & Related papers  (2024-02-26T07:07:00Z)
• Optimal State Manipulation for a Two-Qubit System Driven by Coherent and Incoherent Controls [77.34726150561087]
  State preparation is important for optimal control of two-qubit quantum systems. We exploit two physically different coherent control and optimize the Hilbert-Schmidt target density matrices.
  arXiv  Detail & Related papers  (2023-04-03T10:22:35Z)
• TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasets [118.22975463000928]
  We consider an offline reinforcement learning (RL) setting where the agent need to learn from a dataset collected by rolling out multiple behavior policies. There are two challenges for this setting: 1) The optimal trade-off between optimizing the RL signal and the behavior cloning (BC) signal changes on different states due to the variation of the action coverage induced by different behavior policies. In this paper, we address both challenges by using adaptively weighted reverse Kullback-Leibler (KL) divergence as the BC regularizer based on the TD3 algorithm.
  arXiv  Detail & Related papers  (2022-12-05T09:36:23Z)
• Deep Reinforcement Learning with Embedded LQR Controllers [1.256413718364189]
  We introduce a method that integrates LQR control into the action set, allowing generalization and avoiding fixing the computed control in the replay memory. In all cases, we show that adding LQR control can improve performance, although the effect is more profound if it can be used to augment a discrete action set.
  arXiv  Detail & Related papers  (2021-01-18T17:28:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.